2023-04-28 14:56:47,401 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c847dfce-b710-d479-c833-1a4b351fc988 2023-04-28 14:56:47,413 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-28 14:56:47,426 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-28 14:56:47,427 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c847dfce-b710-d479-c833-1a4b351fc988/cluster_0c9022ae-d6e9-07a9-5c8b-552302ea3460, deleteOnExit=true 2023-04-28 14:56:47,427 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-28 14:56:47,427 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c847dfce-b710-d479-c833-1a4b351fc988/test.cache.data in system properties and HBase conf 2023-04-28 14:56:47,427 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c847dfce-b710-d479-c833-1a4b351fc988/hadoop.tmp.dir in system properties and HBase conf 2023-04-28 14:56:47,428 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c847dfce-b710-d479-c833-1a4b351fc988/hadoop.log.dir in system properties and HBase conf 2023-04-28 14:56:47,428 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c847dfce-b710-d479-c833-1a4b351fc988/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-28 14:56:47,429 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c847dfce-b710-d479-c833-1a4b351fc988/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-28 14:56:47,429 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-28 14:56:47,523 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-28 14:56:47,854 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-28 14:56:47,857 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c847dfce-b710-d479-c833-1a4b351fc988/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-28 14:56:47,857 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c847dfce-b710-d479-c833-1a4b351fc988/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-28 14:56:47,858 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c847dfce-b710-d479-c833-1a4b351fc988/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-28 14:56:47,858 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c847dfce-b710-d479-c833-1a4b351fc988/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-28 14:56:47,858 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c847dfce-b710-d479-c833-1a4b351fc988/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-28 14:56:47,859 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c847dfce-b710-d479-c833-1a4b351fc988/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-28 14:56:47,859 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c847dfce-b710-d479-c833-1a4b351fc988/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-28 14:56:47,859 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c847dfce-b710-d479-c833-1a4b351fc988/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-28 14:56:47,859 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c847dfce-b710-d479-c833-1a4b351fc988/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-28 14:56:47,860 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c847dfce-b710-d479-c833-1a4b351fc988/nfs.dump.dir in system properties and HBase conf 2023-04-28 14:56:47,860 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c847dfce-b710-d479-c833-1a4b351fc988/java.io.tmpdir in system properties and HBase conf 2023-04-28 14:56:47,860 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c847dfce-b710-d479-c833-1a4b351fc988/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-28 14:56:47,860 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c847dfce-b710-d479-c833-1a4b351fc988/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-28 14:56:47,860 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c847dfce-b710-d479-c833-1a4b351fc988/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-28 14:56:48,276 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-28 14:56:48,279 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-28 14:56:48,510 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-28 14:56:48,712 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-28 14:56:48,746 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-28 14:56:48,799 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-28 14:56:48,847 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c847dfce-b710-d479-c833-1a4b351fc988/java.io.tmpdir/Jetty_localhost_localdomain_34187_hdfs____7of3d5/webapp 2023-04-28 14:56:49,004 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:34187 2023-04-28 14:56:49,039 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-28 14:56:49,040 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-28 14:56:49,550 WARN [Listener at localhost.localdomain/44165] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-28 14:56:49,674 WARN [Listener at localhost.localdomain/44165] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-28 14:56:49,700 WARN [Listener at localhost.localdomain/44165] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-28 14:56:49,710 INFO [Listener at localhost.localdomain/44165] log.Slf4jLog(67): jetty-6.1.26 2023-04-28 14:56:49,716 INFO [Listener at localhost.localdomain/44165] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c847dfce-b710-d479-c833-1a4b351fc988/java.io.tmpdir/Jetty_localhost_35001_datanode____epxn92/webapp 2023-04-28 14:56:49,827 INFO [Listener at localhost.localdomain/44165] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35001 2023-04-28 14:56:50,147 WARN [Listener at localhost.localdomain/40193] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-28 14:56:50,517 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x3ca0e033f9ad51b7: Processing first storage report for DS-f9c7ac79-b1ee-4041-97ae-54a791de35e8 from datanode 72d0f6de-cc10-4518-ad17-bf64b7a974d3 2023-04-28 14:56:50,519 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x3ca0e033f9ad51b7: from storage DS-f9c7ac79-b1ee-4041-97ae-54a791de35e8 node DatanodeRegistration(127.0.0.1:44407, datanodeUuid=72d0f6de-cc10-4518-ad17-bf64b7a974d3, infoPort=33111, infoSecurePort=0, ipcPort=40193, storageInfo=lv=-57;cid=testClusterID;nsid=277926357;c=1682693808338), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-28 14:56:50,519 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x3ca0e033f9ad51b7: Processing first storage report for DS-d6e10521-e99e-478e-82a0-955c37941033 from datanode 72d0f6de-cc10-4518-ad17-bf64b7a974d3 2023-04-28 14:56:50,519 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x3ca0e033f9ad51b7: from storage DS-d6e10521-e99e-478e-82a0-955c37941033 node DatanodeRegistration(127.0.0.1:44407, datanodeUuid=72d0f6de-cc10-4518-ad17-bf64b7a974d3, infoPort=33111, infoSecurePort=0, ipcPort=40193, storageInfo=lv=-57;cid=testClusterID;nsid=277926357;c=1682693808338), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-28 14:56:50,538 DEBUG [Listener at localhost.localdomain/40193] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c847dfce-b710-d479-c833-1a4b351fc988 2023-04-28 14:56:50,585 INFO [Listener at localhost.localdomain/40193] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c847dfce-b710-d479-c833-1a4b351fc988/cluster_0c9022ae-d6e9-07a9-5c8b-552302ea3460/zookeeper_0, clientPort=61762, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c847dfce-b710-d479-c833-1a4b351fc988/cluster_0c9022ae-d6e9-07a9-5c8b-552302ea3460/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c847dfce-b710-d479-c833-1a4b351fc988/cluster_0c9022ae-d6e9-07a9-5c8b-552302ea3460/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-28 14:56:50,596 INFO [Listener at localhost.localdomain/40193] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=61762 2023-04-28 14:56:50,603 INFO [Listener at localhost.localdomain/40193] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-28 14:56:50,605 INFO [Listener at localhost.localdomain/40193] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-28 14:56:51,218 INFO [Listener at localhost.localdomain/40193] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106 with version=8 2023-04-28 14:56:51,218 INFO [Listener at localhost.localdomain/40193] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/hbase-staging 2023-04-28 14:56:51,507 INFO [Listener at localhost.localdomain/40193] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-28 14:56:51,873 INFO [Listener at localhost.localdomain/40193] client.ConnectionUtils(127): master/jenkins-hbase17:0 server-side Connection retries=45 2023-04-28 14:56:51,917 INFO [Listener at localhost.localdomain/40193] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-28 14:56:51,917 INFO [Listener at localhost.localdomain/40193] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-28 14:56:51,918 INFO [Listener at localhost.localdomain/40193] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-28 14:56:51,918 INFO [Listener at localhost.localdomain/40193] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-28 14:56:51,918 INFO [Listener at localhost.localdomain/40193] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-28 14:56:52,089 INFO [Listener at localhost.localdomain/40193] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-28 14:56:52,167 DEBUG [Listener at localhost.localdomain/40193] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-28 14:56:52,270 INFO [Listener at localhost.localdomain/40193] ipc.NettyRpcServer(120): Bind to /136.243.18.41:37045 2023-04-28 14:56:52,281 INFO [Listener at localhost.localdomain/40193] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-28 14:56:52,284 INFO [Listener at localhost.localdomain/40193] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-28 14:56:52,304 INFO [Listener at localhost.localdomain/40193] zookeeper.RecoverableZooKeeper(93): Process identifier=master:37045 connecting to ZooKeeper ensemble=127.0.0.1:61762 2023-04-28 14:56:52,348 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:370450x0, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-28 14:56:52,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:37045-0x101d07a98b80000 connected 2023-04-28 14:56:52,383 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-28 14:56:52,384 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-28 14:56:52,387 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-28 14:56:52,395 DEBUG [Listener at localhost.localdomain/40193] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37045 2023-04-28 14:56:52,398 DEBUG [Listener at localhost.localdomain/40193] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37045 2023-04-28 14:56:52,398 DEBUG [Listener at localhost.localdomain/40193] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37045 2023-04-28 14:56:52,402 DEBUG [Listener at localhost.localdomain/40193] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37045 2023-04-28 14:56:52,402 DEBUG [Listener at localhost.localdomain/40193] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37045 2023-04-28 14:56:52,409 INFO [Listener at localhost.localdomain/40193] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106, hbase.cluster.distributed=false 2023-04-28 14:56:52,483 INFO [Listener at localhost.localdomain/40193] client.ConnectionUtils(127): regionserver/jenkins-hbase17:0 server-side Connection retries=45 2023-04-28 14:56:52,483 INFO [Listener at localhost.localdomain/40193] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-28 14:56:52,483 INFO [Listener at localhost.localdomain/40193] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-28 14:56:52,483 INFO [Listener at localhost.localdomain/40193] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-28 14:56:52,484 INFO [Listener at localhost.localdomain/40193] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-28 14:56:52,484 INFO [Listener at localhost.localdomain/40193] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-28 14:56:52,489 INFO [Listener at localhost.localdomain/40193] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-28 14:56:52,492 INFO [Listener at localhost.localdomain/40193] ipc.NettyRpcServer(120): Bind to /136.243.18.41:37181 2023-04-28 14:56:52,495 INFO [Listener at localhost.localdomain/40193] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-28 14:56:52,501 DEBUG [Listener at localhost.localdomain/40193] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-28 14:56:52,502 INFO [Listener at localhost.localdomain/40193] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-28 14:56:52,505 INFO [Listener at localhost.localdomain/40193] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-28 14:56:52,507 INFO [Listener at localhost.localdomain/40193] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:37181 connecting to ZooKeeper ensemble=127.0.0.1:61762 2023-04-28 14:56:52,510 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:371810x0, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-28 14:56:52,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:37181-0x101d07a98b80001 connected 2023-04-28 14:56:52,511 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-28 14:56:52,512 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-28 14:56:52,513 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-28 14:56:52,513 DEBUG [Listener at localhost.localdomain/40193] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37181 2023-04-28 14:56:52,514 DEBUG [Listener at localhost.localdomain/40193] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37181 2023-04-28 14:56:52,514 DEBUG [Listener at localhost.localdomain/40193] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37181 2023-04-28 14:56:52,514 DEBUG [Listener at localhost.localdomain/40193] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37181 2023-04-28 14:56:52,515 DEBUG [Listener at localhost.localdomain/40193] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37181 2023-04-28 14:56:52,517 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase17.apache.org,37045,1682693811371 2023-04-28 14:56:52,524 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-28 14:56:52,525 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase17.apache.org,37045,1682693811371 2023-04-28 14:56:52,541 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-28 14:56:52,541 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-28 14:56:52,542 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-28 14:56:52,542 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-28 14:56:52,543 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase17.apache.org,37045,1682693811371 from backup master directory 2023-04-28 14:56:52,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-28 14:56:52,545 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase17.apache.org,37045,1682693811371 2023-04-28 14:56:52,545 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-28 14:56:52,546 WARN [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-28 14:56:52,546 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase17.apache.org,37045,1682693811371 2023-04-28 14:56:52,549 INFO [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-28 14:56:52,550 INFO [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-28 14:56:52,624 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/hbase.id with ID: 32e64bef-89c2-4ef9-bc0e-8944c420a11e 2023-04-28 14:56:52,668 INFO [master/jenkins-hbase17:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-28 14:56:52,685 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-28 14:56:52,730 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x20638564 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:56:52,760 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f9c4745, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:56:52,780 INFO [master/jenkins-hbase17:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-28 14:56:52,782 INFO [master/jenkins-hbase17:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-28 14:56:52,799 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-28 14:56:52,799 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-28 14:56:52,801 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-28 14:56:52,805 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-28 14:56:52,806 INFO [master/jenkins-hbase17:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-28 14:56:52,835 INFO [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/MasterData/data/master/store-tmp 2023-04-28 14:56:52,859 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 14:56:52,859 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-28 14:56:52,859 INFO [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-28 14:56:52,860 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-28 14:56:52,860 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-28 14:56:52,860 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-28 14:56:52,860 INFO [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-28 14:56:52,860 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-28 14:56:52,862 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/MasterData/WALs/jenkins-hbase17.apache.org,37045,1682693811371 2023-04-28 14:56:52,885 INFO [master/jenkins-hbase17:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase17.apache.org%2C37045%2C1682693811371, suffix=, logDir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/MasterData/WALs/jenkins-hbase17.apache.org,37045,1682693811371, archiveDir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/MasterData/oldWALs, maxLogs=10 2023-04-28 14:56:52,930 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44407,DS-f9c7ac79-b1ee-4041-97ae-54a791de35e8,DISK] 2023-04-28 14:56:52,936 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 14:56:52,987 INFO [master/jenkins-hbase17:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/MasterData/WALs/jenkins-hbase17.apache.org,37045,1682693811371/jenkins-hbase17.apache.org%2C37045%2C1682693811371.1682693812894 2023-04-28 14:56:52,988 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44407,DS-f9c7ac79-b1ee-4041-97ae-54a791de35e8,DISK]] 2023-04-28 14:56:52,989 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-28 14:56:52,989 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 14:56:52,994 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-28 14:56:52,995 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-28 14:56:53,053 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-28 14:56:53,060 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-28 14:56:53,080 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-28 14:56:53,093 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:56:53,099 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-28 14:56:53,101 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-28 14:56:53,120 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-28 14:56:53,124 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-28 14:56:53,125 INFO [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65780533, jitterRate=-0.01979367434978485}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-28 14:56:53,125 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-28 14:56:53,127 INFO [master/jenkins-hbase17:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-28 14:56:53,143 INFO [master/jenkins-hbase17:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-28 14:56:53,144 INFO [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-28 14:56:53,146 INFO [master/jenkins-hbase17:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-28 14:56:53,147 INFO [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-28 14:56:53,172 INFO [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 24 msec 2023-04-28 14:56:53,172 INFO [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-28 14:56:53,193 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-28 14:56:53,198 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-28 14:56:53,225 INFO [master/jenkins-hbase17:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-28 14:56:53,230 INFO [master/jenkins-hbase17:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-28 14:56:53,232 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-28 14:56:53,238 INFO [master/jenkins-hbase17:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-28 14:56:53,243 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-28 14:56:53,246 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-28 14:56:53,247 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-28 14:56:53,248 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-28 14:56:53,258 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-28 14:56:53,262 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-28 14:56:53,262 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-28 14:56:53,262 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-28 14:56:53,262 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase17.apache.org,37045,1682693811371, sessionid=0x101d07a98b80000, setting cluster-up flag (Was=false) 2023-04-28 14:56:53,274 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-28 14:56:53,277 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-28 14:56:53,279 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase17.apache.org,37045,1682693811371 2023-04-28 14:56:53,283 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-28 14:56:53,287 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-28 14:56:53,288 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase17.apache.org,37045,1682693811371 2023-04-28 14:56:53,291 WARN [master/jenkins-hbase17:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.hbase-snapshot/.tmp 2023-04-28 14:56:53,319 INFO [RS:0;jenkins-hbase17:37181] regionserver.HRegionServer(952): ClusterId : 32e64bef-89c2-4ef9-bc0e-8944c420a11e 2023-04-28 14:56:53,323 DEBUG [RS:0;jenkins-hbase17:37181] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-28 14:56:53,327 DEBUG [RS:0;jenkins-hbase17:37181] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-28 14:56:53,327 DEBUG [RS:0;jenkins-hbase17:37181] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-28 14:56:53,330 DEBUG [RS:0;jenkins-hbase17:37181] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-28 14:56:53,331 DEBUG [RS:0;jenkins-hbase17:37181] zookeeper.ReadOnlyZKClient(139): Connect 0x5b182454 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:56:53,336 DEBUG [RS:0;jenkins-hbase17:37181] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@55dfd147, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:56:53,337 DEBUG [RS:0;jenkins-hbase17:37181] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51e04f24, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase17.apache.org/136.243.18.41:0 2023-04-28 14:56:53,368 DEBUG [RS:0;jenkins-hbase17:37181] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase17:37181 2023-04-28 14:56:53,373 INFO [RS:0;jenkins-hbase17:37181] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-28 14:56:53,373 INFO [RS:0;jenkins-hbase17:37181] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-28 14:56:53,373 DEBUG [RS:0;jenkins-hbase17:37181] regionserver.HRegionServer(1023): About to register with Master. 2023-04-28 14:56:53,376 INFO [RS:0;jenkins-hbase17:37181] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase17.apache.org,37045,1682693811371 with isa=jenkins-hbase17.apache.org/136.243.18.41:37181, startcode=1682693812482 2023-04-28 14:56:53,394 DEBUG [RS:0;jenkins-hbase17:37181] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-28 14:56:53,407 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-28 14:56:53,416 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase17:0, corePoolSize=5, maxPoolSize=5 2023-04-28 14:56:53,416 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase17:0, corePoolSize=5, maxPoolSize=5 2023-04-28 14:56:53,416 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase17:0, corePoolSize=5, maxPoolSize=5 2023-04-28 14:56:53,417 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase17:0, corePoolSize=5, maxPoolSize=5 2023-04-28 14:56:53,417 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase17:0, corePoolSize=10, maxPoolSize=10 2023-04-28 14:56:53,417 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-04-28 14:56:53,417 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase17:0, corePoolSize=2, maxPoolSize=2 2023-04-28 14:56:53,417 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-04-28 14:56:53,430 INFO [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682693843430 2023-04-28 14:56:53,432 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-28 14:56:53,433 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-28 14:56:53,434 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-28 14:56:53,440 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-28 14:56:53,444 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-28 14:56:53,450 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:45423, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-28 14:56:53,451 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-28 14:56:53,452 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-28 14:56:53,452 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-28 14:56:53,453 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-28 14:56:53,456 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-28 14:56:53,458 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-28 14:56:53,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:56:53,463 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-28 14:56:53,464 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-28 14:56:53,470 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-28 14:56:53,472 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-28 14:56:53,476 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase17:0:becomeActiveMaster-HFileCleaner.large.0-1682693813475,5,FailOnTimeoutGroup] 2023-04-28 14:56:53,476 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase17:0:becomeActiveMaster-HFileCleaner.small.0-1682693813476,5,FailOnTimeoutGroup] 2023-04-28 14:56:53,477 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-28 14:56:53,477 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-28 14:56:53,480 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-28 14:56:53,480 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-28 14:56:53,484 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-28 14:56:53,485 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-28 14:56:53,485 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106 2023-04-28 14:56:53,496 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 14:56:53,498 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-28 14:56:53,499 DEBUG [RS:0;jenkins-hbase17:37181] regionserver.HRegionServer(2829): Master is not running yet 2023-04-28 14:56:53,499 WARN [RS:0;jenkins-hbase17:37181] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-04-28 14:56:53,501 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/meta/1588230740/info 2023-04-28 14:56:53,502 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-28 14:56:53,503 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:56:53,503 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-28 14:56:53,505 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/meta/1588230740/rep_barrier 2023-04-28 14:56:53,506 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-28 14:56:53,507 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:56:53,507 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-28 14:56:53,509 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/meta/1588230740/table 2023-04-28 14:56:53,509 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-28 14:56:53,510 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:56:53,511 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/meta/1588230740 2023-04-28 14:56:53,512 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/meta/1588230740 2023-04-28 14:56:53,516 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 14:56:53,518 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-28 14:56:53,522 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-28 14:56:53,523 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74042154, jitterRate=0.10331407189369202}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 14:56:53,523 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-28 14:56:53,523 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-28 14:56:53,523 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-28 14:56:53,523 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-28 14:56:53,523 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-28 14:56:53,523 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-28 14:56:53,524 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-28 14:56:53,524 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-28 14:56:53,528 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-28 14:56:53,528 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-28 14:56:53,535 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-28 14:56:53,546 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-28 14:56:53,548 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-28 14:56:53,601 INFO [RS:0;jenkins-hbase17:37181] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase17.apache.org,37045,1682693811371 with isa=jenkins-hbase17.apache.org/136.243.18.41:37181, startcode=1682693812482 2023-04-28 14:56:53,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.ServerManager(394): Registering regionserver=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:53,614 DEBUG [RS:0;jenkins-hbase17:37181] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106 2023-04-28 14:56:53,614 DEBUG [RS:0;jenkins-hbase17:37181] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost.localdomain:44165 2023-04-28 14:56:53,614 DEBUG [RS:0;jenkins-hbase17:37181] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-28 14:56:53,618 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-28 14:56:53,619 DEBUG [RS:0;jenkins-hbase17:37181] zookeeper.ZKUtil(162): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:53,619 WARN [RS:0;jenkins-hbase17:37181] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-28 14:56:53,620 INFO [RS:0;jenkins-hbase17:37181] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-28 14:56:53,620 DEBUG [RS:0;jenkins-hbase17:37181] regionserver.HRegionServer(1947): logDir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/WALs/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:53,620 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase17.apache.org,37181,1682693812482] 2023-04-28 14:56:53,629 DEBUG [RS:0;jenkins-hbase17:37181] zookeeper.ZKUtil(162): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:53,639 DEBUG [RS:0;jenkins-hbase17:37181] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-28 14:56:53,646 INFO [RS:0;jenkins-hbase17:37181] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-28 14:56:53,664 INFO [RS:0;jenkins-hbase17:37181] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-28 14:56:53,666 INFO [RS:0;jenkins-hbase17:37181] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-28 14:56:53,666 INFO [RS:0;jenkins-hbase17:37181] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-28 14:56:53,668 INFO [RS:0;jenkins-hbase17:37181] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-28 14:56:53,673 INFO [RS:0;jenkins-hbase17:37181] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-28 14:56:53,673 DEBUG [RS:0;jenkins-hbase17:37181] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-04-28 14:56:53,674 DEBUG [RS:0;jenkins-hbase17:37181] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-04-28 14:56:53,674 DEBUG [RS:0;jenkins-hbase17:37181] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-04-28 14:56:53,674 DEBUG [RS:0;jenkins-hbase17:37181] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-04-28 14:56:53,674 DEBUG [RS:0;jenkins-hbase17:37181] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-04-28 14:56:53,674 DEBUG [RS:0;jenkins-hbase17:37181] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0, corePoolSize=2, maxPoolSize=2 2023-04-28 14:56:53,674 DEBUG [RS:0;jenkins-hbase17:37181] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-04-28 14:56:53,674 DEBUG [RS:0;jenkins-hbase17:37181] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-04-28 14:56:53,674 DEBUG [RS:0;jenkins-hbase17:37181] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-04-28 14:56:53,674 DEBUG [RS:0;jenkins-hbase17:37181] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-04-28 14:56:53,675 INFO [RS:0;jenkins-hbase17:37181] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-28 14:56:53,675 INFO [RS:0;jenkins-hbase17:37181] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-28 14:56:53,675 INFO [RS:0;jenkins-hbase17:37181] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-28 14:56:53,687 INFO [RS:0;jenkins-hbase17:37181] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-28 14:56:53,689 INFO [RS:0;jenkins-hbase17:37181] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase17.apache.org,37181,1682693812482-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-28 14:56:53,700 DEBUG [jenkins-hbase17:37045] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-28 14:56:53,702 INFO [RS:0;jenkins-hbase17:37181] regionserver.Replication(203): jenkins-hbase17.apache.org,37181,1682693812482 started 2023-04-28 14:56:53,702 INFO [RS:0;jenkins-hbase17:37181] regionserver.HRegionServer(1638): Serving as jenkins-hbase17.apache.org,37181,1682693812482, RpcServer on jenkins-hbase17.apache.org/136.243.18.41:37181, sessionid=0x101d07a98b80001 2023-04-28 14:56:53,702 DEBUG [RS:0;jenkins-hbase17:37181] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-28 14:56:53,702 DEBUG [RS:0;jenkins-hbase17:37181] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:53,702 INFO [PEWorker-2] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase17.apache.org,37181,1682693812482, state=OPENING 2023-04-28 14:56:53,702 DEBUG [RS:0;jenkins-hbase17:37181] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase17.apache.org,37181,1682693812482' 2023-04-28 14:56:53,702 DEBUG [RS:0;jenkins-hbase17:37181] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:56:53,703 DEBUG [RS:0;jenkins-hbase17:37181] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:56:53,704 DEBUG [RS:0;jenkins-hbase17:37181] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-28 14:56:53,704 DEBUG [RS:0;jenkins-hbase17:37181] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-28 14:56:53,704 DEBUG [RS:0;jenkins-hbase17:37181] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:53,704 DEBUG [RS:0;jenkins-hbase17:37181] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase17.apache.org,37181,1682693812482' 2023-04-28 14:56:53,704 DEBUG [RS:0;jenkins-hbase17:37181] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-28 14:56:53,704 DEBUG [RS:0;jenkins-hbase17:37181] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-28 14:56:53,705 DEBUG [RS:0;jenkins-hbase17:37181] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-28 14:56:53,705 INFO [RS:0;jenkins-hbase17:37181] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-28 14:56:53,705 INFO [RS:0;jenkins-hbase17:37181] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-28 14:56:53,708 DEBUG [PEWorker-2] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-28 14:56:53,709 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-28 14:56:53,709 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-28 14:56:53,714 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase17.apache.org,37181,1682693812482}] 2023-04-28 14:56:53,821 INFO [RS:0;jenkins-hbase17:37181] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase17.apache.org%2C37181%2C1682693812482, suffix=, logDir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/WALs/jenkins-hbase17.apache.org,37181,1682693812482, archiveDir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/oldWALs, maxLogs=32 2023-04-28 14:56:53,836 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44407,DS-f9c7ac79-b1ee-4041-97ae-54a791de35e8,DISK] 2023-04-28 14:56:53,840 INFO [RS:0;jenkins-hbase17:37181] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/WALs/jenkins-hbase17.apache.org,37181,1682693812482/jenkins-hbase17.apache.org%2C37181%2C1682693812482.1682693813822 2023-04-28 14:56:53,840 DEBUG [RS:0;jenkins-hbase17:37181] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44407,DS-f9c7ac79-b1ee-4041-97ae-54a791de35e8,DISK]] 2023-04-28 14:56:53,900 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:53,903 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-28 14:56:53,908 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:53038, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-28 14:56:53,919 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-28 14:56:53,919 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-28 14:56:53,923 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase17.apache.org%2C37181%2C1682693812482.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/WALs/jenkins-hbase17.apache.org,37181,1682693812482, archiveDir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/oldWALs, maxLogs=32 2023-04-28 14:56:53,937 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44407,DS-f9c7ac79-b1ee-4041-97ae-54a791de35e8,DISK] 2023-04-28 14:56:53,942 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/WALs/jenkins-hbase17.apache.org,37181,1682693812482/jenkins-hbase17.apache.org%2C37181%2C1682693812482.meta.1682693813924.meta 2023-04-28 14:56:53,943 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44407,DS-f9c7ac79-b1ee-4041-97ae-54a791de35e8,DISK]] 2023-04-28 14:56:53,943 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-28 14:56:53,945 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-28 14:56:53,964 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-28 14:56:53,969 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-28 14:56:53,974 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-28 14:56:53,974 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 14:56:53,974 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-28 14:56:53,975 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-28 14:56:53,977 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-28 14:56:53,979 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/meta/1588230740/info 2023-04-28 14:56:53,979 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/meta/1588230740/info 2023-04-28 14:56:53,979 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-28 14:56:53,980 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:56:53,980 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-28 14:56:53,982 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/meta/1588230740/rep_barrier 2023-04-28 14:56:53,982 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/meta/1588230740/rep_barrier 2023-04-28 14:56:53,982 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-28 14:56:53,983 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:56:53,983 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-28 14:56:53,984 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/meta/1588230740/table 2023-04-28 14:56:53,984 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/meta/1588230740/table 2023-04-28 14:56:53,985 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-28 14:56:53,985 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:56:53,987 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/meta/1588230740 2023-04-28 14:56:53,989 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/meta/1588230740 2023-04-28 14:56:53,993 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 14:56:53,996 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-28 14:56:53,997 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68887654, jitterRate=0.026506036520004272}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 14:56:53,997 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-28 14:56:54,006 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1682693813891 2023-04-28 14:56:54,020 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-28 14:56:54,021 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-28 14:56:54,021 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase17.apache.org,37181,1682693812482, state=OPEN 2023-04-28 14:56:54,023 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-28 14:56:54,023 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-28 14:56:54,027 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-28 14:56:54,027 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase17.apache.org,37181,1682693812482 in 310 msec 2023-04-28 14:56:54,032 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-28 14:56:54,032 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 493 msec 2023-04-28 14:56:54,037 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 703 msec 2023-04-28 14:56:54,038 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1682693814037, completionTime=-1 2023-04-28 14:56:54,038 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-28 14:56:54,038 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-28 14:56:54,094 DEBUG [hconnection-0x1ad43520-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:56:54,097 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:53048, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:56:54,111 INFO [master/jenkins-hbase17:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-28 14:56:54,112 INFO [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682693874112 2023-04-28 14:56:54,112 INFO [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682693934112 2023-04-28 14:56:54,112 INFO [master/jenkins-hbase17:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 73 msec 2023-04-28 14:56:54,132 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase17.apache.org,37045,1682693811371-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-28 14:56:54,132 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase17.apache.org,37045,1682693811371-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-28 14:56:54,132 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase17.apache.org,37045,1682693811371-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-28 14:56:54,134 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase17:37045, period=300000, unit=MILLISECONDS is enabled. 2023-04-28 14:56:54,134 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-28 14:56:54,140 DEBUG [master/jenkins-hbase17:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-28 14:56:54,146 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-28 14:56:54,148 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-28 14:56:54,155 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-28 14:56:54,157 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-28 14:56:54,160 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-28 14:56:54,181 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/hbase/namespace/afac81c8140d02aaf48051ad94c432ec 2023-04-28 14:56:54,184 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/hbase/namespace/afac81c8140d02aaf48051ad94c432ec empty. 2023-04-28 14:56:54,185 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/hbase/namespace/afac81c8140d02aaf48051ad94c432ec 2023-04-28 14:56:54,185 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-28 14:56:54,614 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-28 14:56:54,618 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => afac81c8140d02aaf48051ad94c432ec, NAME => 'hbase:namespace,,1682693814147.afac81c8140d02aaf48051ad94c432ec.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp 2023-04-28 14:56:55,037 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682693814147.afac81c8140d02aaf48051ad94c432ec.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 14:56:55,038 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing afac81c8140d02aaf48051ad94c432ec, disabling compactions & flushes 2023-04-28 14:56:55,038 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682693814147.afac81c8140d02aaf48051ad94c432ec. 2023-04-28 14:56:55,038 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682693814147.afac81c8140d02aaf48051ad94c432ec. 2023-04-28 14:56:55,039 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682693814147.afac81c8140d02aaf48051ad94c432ec. after waiting 0 ms 2023-04-28 14:56:55,039 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682693814147.afac81c8140d02aaf48051ad94c432ec. 2023-04-28 14:56:55,039 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682693814147.afac81c8140d02aaf48051ad94c432ec. 2023-04-28 14:56:55,039 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for afac81c8140d02aaf48051ad94c432ec: 2023-04-28 14:56:55,046 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-28 14:56:55,060 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1682693814147.afac81c8140d02aaf48051ad94c432ec.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682693815049"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682693815049"}]},"ts":"1682693815049"} 2023-04-28 14:56:55,082 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-28 14:56:55,084 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-28 14:56:55,088 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682693815084"}]},"ts":"1682693815084"} 2023-04-28 14:56:55,093 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-28 14:56:55,100 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=afac81c8140d02aaf48051ad94c432ec, ASSIGN}] 2023-04-28 14:56:55,104 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=afac81c8140d02aaf48051ad94c432ec, ASSIGN 2023-04-28 14:56:55,105 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=afac81c8140d02aaf48051ad94c432ec, ASSIGN; state=OFFLINE, location=jenkins-hbase17.apache.org,37181,1682693812482; forceNewPlan=false, retain=false 2023-04-28 14:56:55,256 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=afac81c8140d02aaf48051ad94c432ec, regionState=OPENING, regionLocation=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:55,257 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1682693814147.afac81c8140d02aaf48051ad94c432ec.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682693815256"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682693815256"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682693815256"}]},"ts":"1682693815256"} 2023-04-28 14:56:55,263 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure afac81c8140d02aaf48051ad94c432ec, server=jenkins-hbase17.apache.org,37181,1682693812482}] 2023-04-28 14:56:55,431 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1682693814147.afac81c8140d02aaf48051ad94c432ec. 2023-04-28 14:56:55,432 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => afac81c8140d02aaf48051ad94c432ec, NAME => 'hbase:namespace,,1682693814147.afac81c8140d02aaf48051ad94c432ec.', STARTKEY => '', ENDKEY => ''} 2023-04-28 14:56:55,434 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace afac81c8140d02aaf48051ad94c432ec 2023-04-28 14:56:55,434 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682693814147.afac81c8140d02aaf48051ad94c432ec.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 14:56:55,434 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7894): checking encryption for afac81c8140d02aaf48051ad94c432ec 2023-04-28 14:56:55,435 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7897): checking classloading for afac81c8140d02aaf48051ad94c432ec 2023-04-28 14:56:55,437 INFO [StoreOpener-afac81c8140d02aaf48051ad94c432ec-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region afac81c8140d02aaf48051ad94c432ec 2023-04-28 14:56:55,439 DEBUG [StoreOpener-afac81c8140d02aaf48051ad94c432ec-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/namespace/afac81c8140d02aaf48051ad94c432ec/info 2023-04-28 14:56:55,439 DEBUG [StoreOpener-afac81c8140d02aaf48051ad94c432ec-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/namespace/afac81c8140d02aaf48051ad94c432ec/info 2023-04-28 14:56:55,439 INFO [StoreOpener-afac81c8140d02aaf48051ad94c432ec-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region afac81c8140d02aaf48051ad94c432ec columnFamilyName info 2023-04-28 14:56:55,440 INFO [StoreOpener-afac81c8140d02aaf48051ad94c432ec-1] regionserver.HStore(310): Store=afac81c8140d02aaf48051ad94c432ec/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:56:55,442 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/namespace/afac81c8140d02aaf48051ad94c432ec 2023-04-28 14:56:55,442 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/namespace/afac81c8140d02aaf48051ad94c432ec 2023-04-28 14:56:55,447 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1055): writing seq id for afac81c8140d02aaf48051ad94c432ec 2023-04-28 14:56:55,450 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/namespace/afac81c8140d02aaf48051ad94c432ec/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-28 14:56:55,451 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1072): Opened afac81c8140d02aaf48051ad94c432ec; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62802250, jitterRate=-0.06417354941368103}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-28 14:56:55,451 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(965): Region open journal for afac81c8140d02aaf48051ad94c432ec: 2023-04-28 14:56:55,453 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1682693814147.afac81c8140d02aaf48051ad94c432ec., pid=6, masterSystemTime=1682693815419 2023-04-28 14:56:55,456 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1682693814147.afac81c8140d02aaf48051ad94c432ec. 2023-04-28 14:56:55,456 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1682693814147.afac81c8140d02aaf48051ad94c432ec. 2023-04-28 14:56:55,457 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=afac81c8140d02aaf48051ad94c432ec, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:55,458 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1682693814147.afac81c8140d02aaf48051ad94c432ec.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682693815457"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1682693815457"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682693815457"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682693815457"}]},"ts":"1682693815457"} 2023-04-28 14:56:55,466 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-28 14:56:55,466 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure afac81c8140d02aaf48051ad94c432ec, server=jenkins-hbase17.apache.org,37181,1682693812482 in 200 msec 2023-04-28 14:56:55,469 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-28 14:56:55,470 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=afac81c8140d02aaf48051ad94c432ec, ASSIGN in 366 msec 2023-04-28 14:56:55,471 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-28 14:56:55,472 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682693815472"}]},"ts":"1682693815472"} 2023-04-28 14:56:55,475 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-28 14:56:55,478 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-28 14:56:55,481 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.3300 sec 2023-04-28 14:56:55,561 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-28 14:56:55,563 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-28 14:56:55,564 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-28 14:56:55,606 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-28 14:56:55,624 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-28 14:56:55,631 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 34 msec 2023-04-28 14:56:55,640 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-28 14:56:55,658 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-28 14:56:55,662 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 22 msec 2023-04-28 14:56:55,675 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-28 14:56:55,677 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-28 14:56:55,677 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.131sec 2023-04-28 14:56:55,680 INFO [master/jenkins-hbase17:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-28 14:56:55,681 INFO [master/jenkins-hbase17:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-28 14:56:55,681 INFO [master/jenkins-hbase17:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-28 14:56:55,682 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase17.apache.org,37045,1682693811371-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-28 14:56:55,683 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase17.apache.org,37045,1682693811371-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-28 14:56:55,694 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-28 14:56:55,728 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x7ddc537c to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:56:55,733 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1545112e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:56:55,748 DEBUG [hconnection-0x7019db20-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:56:55,759 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:53062, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:56:55,770 INFO [Listener at localhost.localdomain/40193] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase17.apache.org,37045,1682693811371 2023-04-28 14:56:55,790 INFO [Listener at localhost.localdomain/40193] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=207, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=261, ProcessCount=168, AvailableMemoryMB=5231 2023-04-28 14:56:55,804 DEBUG [Listener at localhost.localdomain/40193] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-28 14:56:55,807 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:51578, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-28 14:56:55,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-28 14:56:55,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.HMaster$4(2112): Client=jenkins//136.243.18.41 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-28 14:56:55,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-28 14:56:55,824 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-28 14:56:55,827 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-28 14:56:55,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(697): Client=jenkins//136.243.18.41 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-28 14:56:55,830 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b 2023-04-28 14:56:55,831 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b empty. 2023-04-28 14:56:55,833 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b 2023-04-28 14:56:55,833 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 14:56:55,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-28 14:56:55,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-28 14:56:56,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-28 14:56:56,254 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-28 14:56:56,258 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => cce278287965607462f53e33830eba7b, NAME => 'TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp 2023-04-28 14:56:56,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-28 14:56:56,675 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 14:56:56,677 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing cce278287965607462f53e33830eba7b, disabling compactions & flushes 2023-04-28 14:56:56,677 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:56:56,677 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:56:56,677 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. after waiting 0 ms 2023-04-28 14:56:56,677 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:56:56,678 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:56:56,678 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:56:56,685 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-28 14:56:56,687 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693816687"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682693816687"}]},"ts":"1682693816687"} 2023-04-28 14:56:56,690 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-28 14:56:56,691 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-28 14:56:56,692 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682693816691"}]},"ts":"1682693816691"} 2023-04-28 14:56:56,694 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-28 14:56:56,697 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=cce278287965607462f53e33830eba7b, ASSIGN}] 2023-04-28 14:56:56,699 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=cce278287965607462f53e33830eba7b, ASSIGN 2023-04-28 14:56:56,701 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=cce278287965607462f53e33830eba7b, ASSIGN; state=OFFLINE, location=jenkins-hbase17.apache.org,37181,1682693812482; forceNewPlan=false, retain=false 2023-04-28 14:56:56,854 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=cce278287965607462f53e33830eba7b, regionState=OPENING, regionLocation=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:56,854 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693816853"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682693816853"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682693816853"}]},"ts":"1682693816853"} 2023-04-28 14:56:56,862 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482}] 2023-04-28 14:56:56,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-28 14:56:57,024 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:56:57,025 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => cce278287965607462f53e33830eba7b, NAME => 'TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.', STARTKEY => '', ENDKEY => ''} 2023-04-28 14:56:57,025 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees cce278287965607462f53e33830eba7b 2023-04-28 14:56:57,025 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 14:56:57,025 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7894): checking encryption for cce278287965607462f53e33830eba7b 2023-04-28 14:56:57,025 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7897): checking classloading for cce278287965607462f53e33830eba7b 2023-04-28 14:56:57,028 INFO [StoreOpener-cce278287965607462f53e33830eba7b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region cce278287965607462f53e33830eba7b 2023-04-28 14:56:57,031 DEBUG [StoreOpener-cce278287965607462f53e33830eba7b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A 2023-04-28 14:56:57,031 DEBUG [StoreOpener-cce278287965607462f53e33830eba7b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A 2023-04-28 14:56:57,033 INFO [StoreOpener-cce278287965607462f53e33830eba7b-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 14:56:57,034 INFO [StoreOpener-cce278287965607462f53e33830eba7b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cce278287965607462f53e33830eba7b columnFamilyName A 2023-04-28 14:56:57,035 INFO [StoreOpener-cce278287965607462f53e33830eba7b-1] regionserver.HStore(310): Store=cce278287965607462f53e33830eba7b/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:56:57,035 INFO [StoreOpener-cce278287965607462f53e33830eba7b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region cce278287965607462f53e33830eba7b 2023-04-28 14:56:57,038 DEBUG [StoreOpener-cce278287965607462f53e33830eba7b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B 2023-04-28 14:56:57,038 DEBUG [StoreOpener-cce278287965607462f53e33830eba7b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B 2023-04-28 14:56:57,038 INFO [StoreOpener-cce278287965607462f53e33830eba7b-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 14:56:57,038 INFO [StoreOpener-cce278287965607462f53e33830eba7b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cce278287965607462f53e33830eba7b columnFamilyName B 2023-04-28 14:56:57,039 INFO [StoreOpener-cce278287965607462f53e33830eba7b-1] regionserver.HStore(310): Store=cce278287965607462f53e33830eba7b/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:56:57,040 INFO [StoreOpener-cce278287965607462f53e33830eba7b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region cce278287965607462f53e33830eba7b 2023-04-28 14:56:57,042 DEBUG [StoreOpener-cce278287965607462f53e33830eba7b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C 2023-04-28 14:56:57,042 DEBUG [StoreOpener-cce278287965607462f53e33830eba7b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C 2023-04-28 14:56:57,042 INFO [StoreOpener-cce278287965607462f53e33830eba7b-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 14:56:57,043 INFO [StoreOpener-cce278287965607462f53e33830eba7b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cce278287965607462f53e33830eba7b columnFamilyName C 2023-04-28 14:56:57,043 INFO [StoreOpener-cce278287965607462f53e33830eba7b-1] regionserver.HStore(310): Store=cce278287965607462f53e33830eba7b/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:56:57,044 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:56:57,047 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b 2023-04-28 14:56:57,048 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b 2023-04-28 14:56:57,053 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 14:56:57,056 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1055): writing seq id for cce278287965607462f53e33830eba7b 2023-04-28 14:56:57,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-28 14:56:57,062 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1072): Opened cce278287965607462f53e33830eba7b; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62410149, jitterRate=-0.07001630961894989}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 14:56:57,062 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(965): Region open journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:56:57,063 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b., pid=11, masterSystemTime=1682693817017 2023-04-28 14:56:57,066 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:56:57,066 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:56:57,067 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=cce278287965607462f53e33830eba7b, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:57,067 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693817067"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1682693817067"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682693817067"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682693817067"}]},"ts":"1682693817067"} 2023-04-28 14:56:57,074 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-28 14:56:57,074 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 in 208 msec 2023-04-28 14:56:57,077 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-28 14:56:57,078 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=cce278287965607462f53e33830eba7b, ASSIGN in 377 msec 2023-04-28 14:56:57,079 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-28 14:56:57,079 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682693817079"}]},"ts":"1682693817079"} 2023-04-28 14:56:57,082 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-28 14:56:57,084 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-28 14:56:57,086 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2640 sec 2023-04-28 14:56:57,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-28 14:56:57,961 INFO [Listener at localhost.localdomain/40193] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-28 14:56:57,967 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x490ff358 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:56:58,063 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ad5f945, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:56:58,068 DEBUG [Listener at localhost.localdomain/40193] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:56:58,073 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:53078, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:56:58,077 DEBUG [Listener at localhost.localdomain/40193] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-28 14:56:58,080 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:51594, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-28 14:56:58,090 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x77443fb0 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:56:58,209 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@142c8aea, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:56:58,210 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x45864842 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:56:58,220 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6362e5a0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:56:58,220 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x4654132d to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:56:58,225 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f537ba4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:56:58,226 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x66ca212c to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:56:58,231 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46b5e40, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:56:58,232 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x1ecabb06 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:56:58,238 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@127a87ef, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:56:58,239 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x3fed686f to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:56:58,246 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@655398b8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:56:58,246 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x3e044e22 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:56:58,270 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a7ef79d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:56:58,270 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x6ab9467e to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:56:58,274 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@776686ab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:56:58,275 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x18f7ec13 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:56:58,278 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@385dd3c5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:56:58,288 DEBUG [hconnection-0x79369e3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:56:58,289 DEBUG [hconnection-0x43d2b8e6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:56:58,290 DEBUG [hconnection-0x32e7e234-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:56:58,291 DEBUG [hconnection-0x26957d3e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:56:58,295 DEBUG [hconnection-0x7d2a217f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:56:58,295 DEBUG [hconnection-0x5be2a28f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:56:58,303 DEBUG [hconnection-0x78372f86-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:56:58,303 DEBUG [hconnection-0x58c2494-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:56:58,303 DEBUG [hconnection-0x37403035-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:56:58,303 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:53088, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:56:58,304 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:53090, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:56:58,304 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:53114, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:56:58,308 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:53092, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:56:58,308 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:53108, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:56:58,311 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:53122, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:56:58,311 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:53134, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:56:58,312 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:53138, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:56:58,314 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:53136, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:56:58,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:56:58,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:56:58,337 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:56:58,337 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:56:58,338 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:56:58,338 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:56:58,342 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:56:58,342 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:56:58,343 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:56:58,343 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:58,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:56:58,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:56:58,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:56:58,344 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:58,344 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:56:58,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:56:58,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:56:58,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:56:58,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:56:58,354 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:56:58,354 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:56:58,354 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:56:58,355 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:56:58,358 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:56:58,358 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:56:58,358 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:56:58,359 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:56:58,368 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cce278287965607462f53e33830eba7b 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-28 14:56:58,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on cce278287965607462f53e33830eba7b 2023-04-28 14:56:58,375 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:56:58,385 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=A 2023-04-28 14:56:58,386 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:56:58,387 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=B 2023-04-28 14:56:58,387 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:56:58,387 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=C 2023-04-28 14:56:58,387 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:56:58,521 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:56:58,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693878491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:58,521 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:56:58,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693878495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:58,530 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:56:58,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693878498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:58,531 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:56:58,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693878522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:58,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693818476, max timestamp=1682693818497Num uniques 859; 2023-04-28 14:56:58,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682693818476, max timestamp=1682693818497Num uniques 859; 2023-04-28 14:56:58,537 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-28 14:56:58,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682693818476, max timestamp=1682693818497Num uniques 859; 2023-04-28 14:56:58,539 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-28 14:56:58,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:56:58,568 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/4fdf12af22864055807f1492dadcd192 2023-04-28 14:56:58,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682693818634, max timestamp=1682693818643Num uniques 1018; 2023-04-28 14:56:58,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-28 14:56:58,664 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682693818634, max timestamp=1682693818643Num uniques 1009; 2023-04-28 14:56:58,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-28 14:56:58,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682693818634, max timestamp=1682693818643Num uniques 1009; 2023-04-28 14:56:58,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-28 14:56:58,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682693818677, max timestamp=1682693818724Num uniques 1027; 2023-04-28 14:56:58,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682693818677, max timestamp=1682693818724Num uniques 1027; 2023-04-28 14:56:58,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-28 14:56:58,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-28 14:56:58,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682693818677, max timestamp=1682693818724Num uniques 1027; 2023-04-28 14:56:58,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-28 14:56:58,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:56:58,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:56:58,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693878772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:58,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693878771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:58,778 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:56:58,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693878774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:58,779 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:56:58,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693878774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:58,780 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:56:58,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693878777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:58,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:56:58,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693878881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:58,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:56:58,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693878881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:58,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:56:58,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:56:58,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693878883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:58,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:56:58,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693878884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:58,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693878882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,086 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:56:59,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693879086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:56:59,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693879088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,090 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:56:59,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693879088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,091 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:56:59,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693879088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:56:59,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693879092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,108 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/59f60be6f15c4048bbb98871bbf32bd5 2023-04-28 14:56:59,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:56:59,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693879389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:56:59,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:56:59,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693879395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693879395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:56:59,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693879396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,399 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:56:59,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693879396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,495 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-28 14:56:59,613 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/5de7ffe8dd4a42328e330523983e791f 2023-04-28 14:56:59,645 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-28 14:56:59,648 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-28 14:56:59,648 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/4fdf12af22864055807f1492dadcd192 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/4fdf12af22864055807f1492dadcd192 2023-04-28 14:56:59,649 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-28 14:56:59,668 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/4fdf12af22864055807f1492dadcd192, entries=150, sequenceid=11, filesize=11.6 K 2023-04-28 14:56:59,677 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/59f60be6f15c4048bbb98871bbf32bd5 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/59f60be6f15c4048bbb98871bbf32bd5 2023-04-28 14:56:59,701 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/59f60be6f15c4048bbb98871bbf32bd5, entries=150, sequenceid=11, filesize=11.6 K 2023-04-28 14:56:59,704 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/5de7ffe8dd4a42328e330523983e791f as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/5de7ffe8dd4a42328e330523983e791f 2023-04-28 14:56:59,741 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/5de7ffe8dd4a42328e330523983e791f, entries=150, sequenceid=11, filesize=11.6 K 2023-04-28 14:56:59,744 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for cce278287965607462f53e33830eba7b in 1376ms, sequenceid=11, compaction requested=false 2023-04-28 14:56:59,746 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:56:59,746 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:56:59,746 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:56:59,747 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:56:59,747 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,747 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:56:59,747 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:56:59,750 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:56:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:56:59,752 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:56:59,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:56:59,753 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:56:59,753 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:56:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:56:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:56:59,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:56:59,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:56:59,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3727be3c[Count = 0] remaining members to acquire global barrier 2023-04-28 14:56:59,756 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:56:59,756 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:56:59,759 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:56:59,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:56:59,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:56:59,760 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:56:59,760 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:56:59,760 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:56:59,766 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:56:59,766 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:56:59,766 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:56:59,772 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,772 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:56:59,773 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:56:59,773 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:56:59,773 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:56:59,773 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:56:59,773 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:56:59,773 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:56:59,782 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:56:59,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:56:59,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:56:59,783 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:56:59,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:56:59,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:56:59,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:56:59,784 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:56:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:56:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:56:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:56:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:56:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:56:59,786 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:56:59,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:56:59,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:56:59,789 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:56:59,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,807 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,808 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:56:59,808 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:56:59,808 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,808 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:56:59,808 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:56:59,808 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:56:59,808 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:56:59,808 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:56:59,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:56:59,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:56:59,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:56:59,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:56:59,808 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:56:59,812 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:56:59,812 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:56:59,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:56:59,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:56:59,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on cce278287965607462f53e33830eba7b 2023-04-28 14:56:59,905 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cce278287965607462f53e33830eba7b 3/3 column families, dataSize=140.89 KB heapSize=398.53 KB 2023-04-28 14:56:59,905 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=A 2023-04-28 14:56:59,905 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:56:59,905 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=B 2023-04-28 14:56:59,906 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:56:59,906 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=C 2023-04-28 14:56:59,906 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:56:59,915 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:56:59,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:56:59,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:56:59,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:56:59,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:56:59,947 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:56:59,947 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:56:59,950 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:56:59,951 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:56:59,954 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:56:59,954 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:56:59,955 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:56:59,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:56:59,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:56:59,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:56:59,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:56:59,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:56:59,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:56:59,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:56:59,958 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,959 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,959 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:56:59,964 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:56:59,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:56:59,967 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:56:59,967 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:56:59,968 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:56:59,970 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:56:59,970 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:56:59,970 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:56:59,970 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:56:59,970 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:56:59,970 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:56:59,985 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:56:59,985 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:56:59,985 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,985 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:56:59,985 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:56:59,985 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/1bfeecfa7d8b4c41aa8b36b1ce397501 2023-04-28 14:56:59,987 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,987 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:56:59,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:56:59,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:56:59,988 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:56:59,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693879931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,988 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:56:59,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693879935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,989 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:56:59,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693879954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:56:59,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693879960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,990 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:56:59,990 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:56:59,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:56:59,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:56:59,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:56:59,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:56:59,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:56:59,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60d62af8[Count = 0] remaining members to acquire global barrier 2023-04-28 14:56:59,994 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:56:59,994 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:56:59,996 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:56:59,996 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:56:59,996 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:56:59,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:56:59,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:56:59,997 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:56:59,997 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:56:59,997 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:00,000 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:00,000 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:00,000 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:00,003 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:00,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:00,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:00,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:00,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:00,008 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:00,008 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:00,008 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:00,008 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:00,009 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:00,015 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,015 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,016 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:00,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:00,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:00,016 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:00,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:00,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:00,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:00,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:00,018 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:00,018 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:00,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693879990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:00,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:00,028 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,028 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:00,028 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:00,028 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,029 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:00,029 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:00,029 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:00,029 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:00,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:00,029 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,029 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:00,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:00,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:00,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:00,029 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:00,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:00,029 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:00,058 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/2f7c14d7b8714d7599f10a54db4afaaf 2023-04-28 14:57:00,101 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:00,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693880092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,102 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:00,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693880092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:00,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693880094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:00,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693880098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:00,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693880124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,130 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:00,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:00,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:00,151 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/9c64a928e5ed41788ef3a4e1737c710e 2023-04-28 14:57:00,167 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/1bfeecfa7d8b4c41aa8b36b1ce397501 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/1bfeecfa7d8b4c41aa8b36b1ce397501 2023-04-28 14:57:00,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:00,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:00,184 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:00,184 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:00,184 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:00,184 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:00,185 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,185 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:00,186 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:00,186 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:00,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:00,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:00,187 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,187 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:00,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:00,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:00,188 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:00,191 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/1bfeecfa7d8b4c41aa8b36b1ce397501, entries=200, sequenceid=82, filesize=13.9 K 2023-04-28 14:57:00,191 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:00,192 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:00,192 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:00,192 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:00,193 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:00,194 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:00,194 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:00,194 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/2f7c14d7b8714d7599f10a54db4afaaf as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/2f7c14d7b8714d7599f10a54db4afaaf 2023-04-28 14:57:00,196 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:00,197 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:00,197 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:00,197 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:00,202 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:00,202 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:00,202 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,202 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:00,202 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:00,205 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,205 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:00,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:00,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:00,205 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:00,205 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:00,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:00,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:00,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:00,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:00,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:00,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45623fe2[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:00,209 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:00,209 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:00,210 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:00,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:00,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:00,210 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:00,210 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:00,210 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:00,211 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,211 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:00,214 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:00,214 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:00,214 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:00,214 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,219 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/2f7c14d7b8714d7599f10a54db4afaaf, entries=150, sequenceid=82, filesize=11.6 K 2023-04-28 14:57:00,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:00,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:00,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:00,226 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/9c64a928e5ed41788ef3a4e1737c710e as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/9c64a928e5ed41788ef3a4e1737c710e 2023-04-28 14:57:00,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:00,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:00,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:00,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:00,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:00,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:00,231 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:00,232 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:00,232 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:00,232 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:00,232 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:00,234 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,234 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:00,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,234 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:00,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:00,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:00,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:00,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:00,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,236 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:00,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:00,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:00,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:00,240 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:00,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,248 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:00,249 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:00,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:00,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:00,250 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,250 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:00,250 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:00,250 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:00,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:00,250 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:00,250 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:00,250 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,250 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:00,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:00,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:00,255 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:00,255 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:00,256 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/9c64a928e5ed41788ef3a4e1737c710e, entries=150, sequenceid=82, filesize=11.6 K 2023-04-28 14:57:00,262 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~398.48 KB/408048, currentSize=40.25 KB/41220 for cce278287965607462f53e33830eba7b in 358ms, sequenceid=82, compaction requested=false 2023-04-28 14:57:00,262 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:00,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on cce278287965607462f53e33830eba7b 2023-04-28 14:57:00,312 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cce278287965607462f53e33830eba7b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 14:57:00,313 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=A 2023-04-28 14:57:00,313 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:00,313 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=B 2023-04-28 14:57:00,313 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:00,313 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=C 2023-04-28 14:57:00,313 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:00,356 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:00,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:00,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:00,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:00,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:00,362 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:00,362 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:00,363 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:00,363 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:00,363 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/6da170faf0be4c0fa7fbda6902573263 2023-04-28 14:57:00,363 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,363 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:00,364 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:00,365 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,365 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,365 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:00,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:00,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:00,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:00,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:00,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:00,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:00,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:00,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:00,370 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:00,370 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:00,371 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:00,378 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:00,378 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:00,378 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:00,378 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:00,378 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:00,378 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:00,381 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:00,381 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:00,381 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,381 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:00,381 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:00,384 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,384 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:00,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:00,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:00,385 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:00,385 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:00,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:00,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:00,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:00,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:00,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:00,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@386d1a01[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:00,389 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:00,389 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:00,390 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:00,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:00,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:00,390 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:00,390 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:00,390 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:00,390 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,391 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:00,399 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:00,399 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:00,399 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:00,399 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:00,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:00,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:00,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:00,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:00,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:00,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:00,407 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:00,407 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:00,408 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:00,408 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:00,408 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:00,410 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,410 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:00,410 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:00,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:00,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,412 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:00,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:00,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:00,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:00,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:00,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,425 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:00,429 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/5b593c12ec764220a541d668e587e298 2023-04-28 14:57:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,436 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,436 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:00,436 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:00,436 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:00,436 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:00,436 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,437 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:00,437 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:00,437 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,437 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:00,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:00,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:00,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:00,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:00,439 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:00,439 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:00,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:00,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:00,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693820408, max timestamp=1682693820460Num uniques 859; 2023-04-28 14:57:00,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:57:00,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693820408, max timestamp=1682693820460Num uniques 859; 2023-04-28 14:57:00,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:57:00,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682693820408, max timestamp=1682693820457Num uniques 859; 2023-04-28 14:57:00,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-28 14:57:00,497 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/0d7c7c781ab4410bb36c6f27dbbc321a 2023-04-28 14:57:00,507 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/6da170faf0be4c0fa7fbda6902573263 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/6da170faf0be4c0fa7fbda6902573263 2023-04-28 14:57:00,518 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/6da170faf0be4c0fa7fbda6902573263, entries=250, sequenceid=93, filesize=16.2 K 2023-04-28 14:57:00,523 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/5b593c12ec764220a541d668e587e298 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/5b593c12ec764220a541d668e587e298 2023-04-28 14:57:00,539 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/5b593c12ec764220a541d668e587e298, entries=150, sequenceid=93, filesize=11.6 K 2023-04-28 14:57:00,543 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:00,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:00,550 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/0d7c7c781ab4410bb36c6f27dbbc321a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/0d7c7c781ab4410bb36c6f27dbbc321a 2023-04-28 14:57:00,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:00,562 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/0d7c7c781ab4410bb36c6f27dbbc321a, entries=150, sequenceid=93, filesize=11.6 K 2023-04-28 14:57:00,565 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=118.13 KB/120965 for cce278287965607462f53e33830eba7b in 253ms, sequenceid=93, compaction requested=true 2023-04-28 14:57:00,565 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:00,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:00,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:00,568 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:00,568 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:00,569 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:00,569 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:00,570 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,570 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:00,571 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,571 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:00,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:00,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:00,571 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,571 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:00,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:00,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:00,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:00,572 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:00,574 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:00,574 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:00,574 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:00,574 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:00,579 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:00,580 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:00,580 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:00,580 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:57:00,580 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:00,580 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:57:00,580 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:00,580 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:57:00,581 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:57:00,581 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:00,583 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cce278287965607462f53e33830eba7b 3/3 column families, dataSize=118.13 KB heapSize=334.46 KB 2023-04-28 14:57:00,584 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=A 2023-04-28 14:57:00,584 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:00,584 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=B 2023-04-28 14:57:00,584 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:00,584 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=C 2023-04-28 14:57:00,584 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:00,587 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:57:00,588 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42678 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:57:00,589 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): cce278287965607462f53e33830eba7b/B is initiating minor compaction (all files) 2023-04-28 14:57:00,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on cce278287965607462f53e33830eba7b 2023-04-28 14:57:00,589 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:00,589 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cce278287965607462f53e33830eba7b/B in TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:00,590 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/59f60be6f15c4048bbb98871bbf32bd5, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/2f7c14d7b8714d7599f10a54db4afaaf, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/5b593c12ec764220a541d668e587e298] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp, totalSize=34.8 K 2023-04-28 14:57:00,590 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): cce278287965607462f53e33830eba7b/A is initiating minor compaction (all files) 2023-04-28 14:57:00,590 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cce278287965607462f53e33830eba7b/A in TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:00,590 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/4fdf12af22864055807f1492dadcd192, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/1bfeecfa7d8b4c41aa8b36b1ce397501, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/6da170faf0be4c0fa7fbda6902573263] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp, totalSize=41.7 K 2023-04-28 14:57:00,591 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 59f60be6f15c4048bbb98871bbf32bd5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1682693818345 2023-04-28 14:57:00,595 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 4fdf12af22864055807f1492dadcd192, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1682693818345 2023-04-28 14:57:00,600 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 2f7c14d7b8714d7599f10a54db4afaaf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682693818744 2023-04-28 14:57:00,604 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 1bfeecfa7d8b4c41aa8b36b1ce397501, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682693818744 2023-04-28 14:57:00,604 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 5b593c12ec764220a541d668e587e298, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1682693819934 2023-04-28 14:57:00,605 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 6da170faf0be4c0fa7fbda6902573263, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1682693819934 2023-04-28 14:57:00,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:00,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693880613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,640 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:00,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693880626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,642 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): cce278287965607462f53e33830eba7b#A#compaction#10 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:00,662 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): cce278287965607462f53e33830eba7b#B#compaction#11 average throughput is 0.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:00,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:00,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693880632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:00,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693880635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:00,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693880638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,675 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=37.62 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/2130444c2f154bd7af647c88371269cd 2023-04-28 14:57:00,730 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/cdd79dacfa474ca98d4a23c45794bc7c as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/cdd79dacfa474ca98d4a23c45794bc7c 2023-04-28 14:57:00,736 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/462d01412a0a4aa8877e9fe946b49fed 2023-04-28 14:57:00,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:00,752 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:00,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693880741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693880742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,763 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cce278287965607462f53e33830eba7b/B of cce278287965607462f53e33830eba7b into cdd79dacfa474ca98d4a23c45794bc7c(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:00,763 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:00,764 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b., storeName=cce278287965607462f53e33830eba7b/B, priority=13, startTime=1682693820580; duration=0sec 2023-04-28 14:57:00,765 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:57:00,765 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:57:00,769 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:57:00,769 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): cce278287965607462f53e33830eba7b/C is initiating minor compaction (all files) 2023-04-28 14:57:00,769 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cce278287965607462f53e33830eba7b/C in TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:00,769 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/5de7ffe8dd4a42328e330523983e791f, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/9c64a928e5ed41788ef3a4e1737c710e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/0d7c7c781ab4410bb36c6f27dbbc321a] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp, totalSize=34.8 K 2023-04-28 14:57:00,770 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 5de7ffe8dd4a42328e330523983e791f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1682693818345 2023-04-28 14:57:00,771 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 9c64a928e5ed41788ef3a4e1737c710e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682693818744 2023-04-28 14:57:00,772 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 0d7c7c781ab4410bb36c6f27dbbc321a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1682693819934 2023-04-28 14:57:00,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:00,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693880767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:00,787 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:00,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693880775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693880767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,790 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/2e44bde400f04fa281c4d11d0e9c90b9 2023-04-28 14:57:00,799 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/2130444c2f154bd7af647c88371269cd as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/2130444c2f154bd7af647c88371269cd 2023-04-28 14:57:00,799 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): cce278287965607462f53e33830eba7b#C#compaction#14 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:00,815 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/2130444c2f154bd7af647c88371269cd, entries=200, sequenceid=127, filesize=13.9 K 2023-04-28 14:57:00,822 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/462d01412a0a4aa8877e9fe946b49fed as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/462d01412a0a4aa8877e9fe946b49fed 2023-04-28 14:57:00,838 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/462d01412a0a4aa8877e9fe946b49fed, entries=150, sequenceid=127, filesize=11.6 K 2023-04-28 14:57:00,841 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/2e44bde400f04fa281c4d11d0e9c90b9 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/2e44bde400f04fa281c4d11d0e9c90b9 2023-04-28 14:57:00,855 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/8f55aa855e0949a88d22561c2a4bc65b as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/8f55aa855e0949a88d22561c2a4bc65b 2023-04-28 14:57:00,864 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/2e44bde400f04fa281c4d11d0e9c90b9, entries=150, sequenceid=127, filesize=11.6 K 2023-04-28 14:57:00,868 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~118.13 KB/120965, heapSize ~334.41 KB/342440, currentSize=60.38 KB/61830 for cce278287965607462f53e33830eba7b in 284ms, sequenceid=127, compaction requested=false 2023-04-28 14:57:00,868 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:00,868 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:00,868 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:00,869 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:00,869 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,869 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:00,869 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:00,872 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:00,872 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:00,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:00,873 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:00,873 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:00,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:00,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:00,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:00,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:00,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:00,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3231956e[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:00,878 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:00,878 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:00,879 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:00,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:00,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:00,880 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:00,880 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,880 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:00,880 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:00,880 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:00,881 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cce278287965607462f53e33830eba7b/C of cce278287965607462f53e33830eba7b into 8f55aa855e0949a88d22561c2a4bc65b(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:00,881 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:00,881 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b., storeName=cce278287965607462f53e33830eba7b/C, priority=13, startTime=1682693820581; duration=0sec 2023-04-28 14:57:00,882 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,887 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:00,882 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:00,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:00,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:00,887 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:00,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:00,887 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:00,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:00,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:00,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:00,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:00,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:00,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:00,898 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:00,898 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:00,898 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:00,898 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:00,898 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:00,902 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,902 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,902 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:00,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:00,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:00,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:00,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:00,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:00,904 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:00,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:00,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:00,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,906 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:00,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:00,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,915 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:00,915 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:00,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:00,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:00,915 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,916 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:00,916 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:00,916 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:00,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:00,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:00,918 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:00,918 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:00,916 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:00,919 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:00,919 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:00,919 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:00,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:00,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on cce278287965607462f53e33830eba7b 2023-04-28 14:57:00,961 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cce278287965607462f53e33830eba7b 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-28 14:57:00,962 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=A 2023-04-28 14:57:00,962 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:00,962 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=B 2023-04-28 14:57:00,962 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:00,962 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=C 2023-04-28 14:57:00,962 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:01,018 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:01,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:01,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:01,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:01,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:01,045 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:01,045 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:01,046 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:01,046 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:01,047 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,047 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,049 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,049 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:01,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:01,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:01,049 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,049 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:01,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:01,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,050 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:01,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:01,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693881036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:01,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693881039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,054 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:01,054 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:01,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:01,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693881042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,054 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:01,054 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:01,058 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:01,059 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:01,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693881045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:01,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693881052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,063 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:01,063 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:01,063 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:01,064 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:01,064 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:01,064 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:01,065 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:01,065 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:01,065 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,065 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:01,065 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:01,067 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,067 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,067 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:01,070 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:01,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:01,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:01,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:01,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:01,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:01,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63e1e83a[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:01,073 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:01,073 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,074 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,074 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:01,074 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:01,074 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:01,075 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,075 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:01,076 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:01,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:01,076 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:01,076 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:01,076 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:01,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:01,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:01,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:01,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:01,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:01,080 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:01,080 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:01,080 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:01,080 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:01,083 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:01,086 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,086 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:01,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,086 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:01,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:01,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:01,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:01,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:01,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,091 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:01,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:01,093 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,098 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:01,098 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:01,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:01,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:01,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:01,100 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:01,100 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,100 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,100 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,100 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,102 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,102 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:01,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:01,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:01,104 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:01,104 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:01,108 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/b3b3655f41e44f1986c99252981c1a8b as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/b3b3655f41e44f1986c99252981c1a8b 2023-04-28 14:57:01,123 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cce278287965607462f53e33830eba7b/A of cce278287965607462f53e33830eba7b into b3b3655f41e44f1986c99252981c1a8b(size=11.7 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:01,125 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:01,125 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b., storeName=cce278287965607462f53e33830eba7b/A, priority=13, startTime=1682693820571; duration=0sec 2023-04-28 14:57:01,125 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:01,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:01,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693881153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,159 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:01,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693881158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,164 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:01,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693881161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:01,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:01,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693881161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693881162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,205 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:01,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:01,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:01,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:01,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:01,217 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:01,217 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:01,218 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:01,218 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:01,219 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,219 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,220 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,220 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:01,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:01,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:01,220 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,220 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:01,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:01,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,221 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:01,221 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:01,221 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:01,221 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:01,221 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:01,222 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:01,223 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:01,223 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:01,223 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:01,223 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:01,223 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:01,223 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:01,223 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:01,223 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:01,223 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,223 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:01,223 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:01,225 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,225 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:01,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:01,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:01,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:01,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,226 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,227 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:01,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:01,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:01,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37a8468b[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:01,228 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:01,228 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,229 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,229 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:01,229 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:01,230 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:01,230 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,230 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:01,230 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,230 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:01,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:01,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:01,231 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:01,231 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:01,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:01,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:01,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:01,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:01,235 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:01,235 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:01,235 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:01,235 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:01,235 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:01,237 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,237 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:01,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,237 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:01,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:01,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:01,239 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,242 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,245 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:01,245 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:01,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:01,245 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:01,246 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:01,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:01,246 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:01,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:01,246 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,251 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,251 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,251 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,251 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,250 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:01,251 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:01,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:01,351 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:01,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:01,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:01,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:01,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:01,359 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:01,359 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:01,359 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:01,360 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:01,360 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,360 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,361 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:01,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:01,361 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:01,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:01,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,363 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:01,364 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,364 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:01,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:01,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693881361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,364 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:01,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693881362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,365 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:01,365 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:01,365 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:01,365 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:01,366 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:01,367 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:01,367 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:01,367 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:01,367 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:01,367 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:01,367 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:01,367 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:01,367 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:01,367 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,367 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:01,367 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:01,369 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:01,369 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:01,370 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:01,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693881367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,370 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:01,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:01,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693881368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,370 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,371 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:01,370 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:01,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693881368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:01,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:01,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6745ef6d[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:01,372 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:01,372 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,373 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,374 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:01,374 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:01,374 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:01,374 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,374 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:01,375 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,375 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:01,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:01,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:01,375 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:01,375 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:01,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:01,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:01,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:01,384 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:01,384 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:01,384 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:01,384 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:01,384 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:01,386 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,386 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:01,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,386 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:01,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:01,388 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,389 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,395 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:01,395 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:01,395 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:01,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:01,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:01,395 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:01,395 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,395 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,395 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,396 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,396 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:01,396 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:01,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:01,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:01,396 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,397 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,420 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/24c860bb48ce40c9b305b336862cb493 2023-04-28 14:57:01,497 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:01,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:01,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:01,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:01,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:01,504 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-28 14:57:01,505 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-28 14:57:01,505 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:01,505 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:01,506 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:01,506 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:01,507 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,507 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,508 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,508 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,508 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:01,510 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:01,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:01,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:01,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:01,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,512 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:01,512 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:01,512 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:01,512 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:01,512 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:01,513 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:01,513 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:01,513 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:01,513 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:01,514 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:01,514 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:01,514 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:01,514 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:01,514 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:01,514 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,514 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:01,514 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:01,516 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,516 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:01,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:01,516 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,516 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:01,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:01,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:01,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:01,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:01,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59e19c0e[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:01,518 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:01,518 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,519 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,520 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:01,520 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:01,520 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:01,520 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,520 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:01,521 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:01,521 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:01,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:01,521 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:01,522 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:01,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:01,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:01,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:01,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:01,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:01,528 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:01,528 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:01,528 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:01,528 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:01,528 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:01,530 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,531 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:01,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,531 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:01,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:01,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:01,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:01,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:01,533 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:01,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,536 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:01,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,539 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:01,539 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:01,539 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:01,539 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:01,540 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,540 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,540 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:01,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:01,541 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,541 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,541 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:01,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:01,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:01,542 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:01,542 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:01,642 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:01,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:01,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:01,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:01,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:01,666 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:01,666 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:01,667 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:01,667 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:01,668 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,668 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:01,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693881667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:01,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693881668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,671 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:01,671 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:01,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:01,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,672 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,672 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:01,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:01,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,672 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:01,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:01,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:01,673 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:01,673 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:01,675 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:01,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:01,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693881676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:01,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693881676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,680 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:01,680 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:01,680 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:01,680 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:01,681 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:01,682 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:01,682 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:01,682 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:01,682 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,682 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:01,682 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:01,682 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:01,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693881681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,683 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,684 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,684 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,684 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:01,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:01,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:01,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:01,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:01,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:01,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:01,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35ad0f30[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:01,686 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:01,686 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,687 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,687 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:01,687 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:01,687 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:01,688 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:01,688 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:01,688 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:01,689 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,689 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:01,689 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:01,689 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:01,689 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:01,689 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:01,689 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:01,689 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:01,695 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,695 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,695 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:01,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:01,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:01,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:01,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:01,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:01,696 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:01,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,699 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:01,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,701 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,701 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:01,701 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,701 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:01,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:01,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:01,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:01,701 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:01,701 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:01,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:01,701 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,702 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:01,702 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,702 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:01,702 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,702 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:01,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:01,802 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:01,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:01,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:01,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:01,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:01,806 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:01,806 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:01,807 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:01,807 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:01,808 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,808 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,808 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:01,808 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:01,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:01,809 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,809 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:01,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:01,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,810 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:01,810 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:01,810 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:01,810 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:01,811 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:01,811 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:01,811 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:01,811 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:01,812 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:01,812 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:01,812 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:01,812 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:01,812 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:01,812 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,812 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:01,812 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:01,813 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,813 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:01,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:01,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:01,814 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,814 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:01,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:01,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:01,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:01,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40449b10[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:01,815 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:01,815 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,816 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,816 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:01,816 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:01,816 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:01,817 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,817 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:01,818 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:01,818 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:01,818 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:01,818 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:01,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:01,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:01,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:01,825 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:01,825 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:01,825 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:01,825 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:01,825 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:01,827 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,827 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:01,827 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:01,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:01,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:01,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:01,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:01,828 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:01,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,829 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:01,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:01,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,832 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:01,832 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:01,832 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,832 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,832 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,832 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:01,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:01,832 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,832 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:01,833 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:01,833 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:01,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:01,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:01,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:01,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:01,835 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:01,835 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:01,852 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/473b1abc37f649598dd6753d62fbafe6 2023-04-28 14:57:01,895 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/61579ac381414e929dfca6576e6704ae 2023-04-28 14:57:01,905 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/24c860bb48ce40c9b305b336862cb493 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/24c860bb48ce40c9b305b336862cb493 2023-04-28 14:57:01,915 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/24c860bb48ce40c9b305b336862cb493, entries=250, sequenceid=143, filesize=16.4 K 2023-04-28 14:57:01,918 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/473b1abc37f649598dd6753d62fbafe6 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/473b1abc37f649598dd6753d62fbafe6 2023-04-28 14:57:01,935 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:01,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:01,936 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/473b1abc37f649598dd6753d62fbafe6, entries=150, sequenceid=143, filesize=11.8 K 2023-04-28 14:57:01,943 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/61579ac381414e929dfca6576e6704ae as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/61579ac381414e929dfca6576e6704ae 2023-04-28 14:57:01,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:01,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:01,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:01,962 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:01,962 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:01,963 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:01,964 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:01,965 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,965 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,966 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/61579ac381414e929dfca6576e6704ae, entries=150, sequenceid=143, filesize=11.8 K 2023-04-28 14:57:01,967 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,967 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:01,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:01,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:01,968 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for cce278287965607462f53e33830eba7b in 1007ms, sequenceid=143, compaction requested=true 2023-04-28 14:57:01,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,968 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:01,968 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:01,968 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:01,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:01,968 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:57:01,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:01,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:01,968 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:57:01,969 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:01,969 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:57:01,971 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-28 14:57:01,971 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 43031 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:57:01,972 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): cce278287965607462f53e33830eba7b/A is initiating minor compaction (all files) 2023-04-28 14:57:01,972 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cce278287965607462f53e33830eba7b/A in TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:01,972 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/b3b3655f41e44f1986c99252981c1a8b, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/2130444c2f154bd7af647c88371269cd, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/24c860bb48ce40c9b305b336862cb493] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp, totalSize=42.0 K 2023-04-28 14:57:01,973 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting b3b3655f41e44f1986c99252981c1a8b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1682693819934 2023-04-28 14:57:01,973 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 2130444c2f154bd7af647c88371269cd, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1682693820535 2023-04-28 14:57:01,974 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 24c860bb48ce40c9b305b336862cb493, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1682693820619 2023-04-28 14:57:01,980 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:57:01,980 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:01,983 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:01,983 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:01,983 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:01,986 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:01,987 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:01,987 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:01,988 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35911 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:57:01,988 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): cce278287965607462f53e33830eba7b/B is initiating minor compaction (all files) 2023-04-28 14:57:01,988 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cce278287965607462f53e33830eba7b/B in TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:01,988 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/cdd79dacfa474ca98d4a23c45794bc7c, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/462d01412a0a4aa8877e9fe946b49fed, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/473b1abc37f649598dd6753d62fbafe6] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp, totalSize=35.1 K 2023-04-28 14:57:01,989 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting cdd79dacfa474ca98d4a23c45794bc7c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1682693819934 2023-04-28 14:57:01,990 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 462d01412a0a4aa8877e9fe946b49fed, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1682693820535 2023-04-28 14:57:01,991 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:01,992 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:01,992 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cce278287965607462f53e33830eba7b 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-28 14:57:01,992 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=A 2023-04-28 14:57:01,993 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:01,993 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=B 2023-04-28 14:57:01,993 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:01,993 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=C 2023-04-28 14:57:01,993 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:01,996 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): cce278287965607462f53e33830eba7b#A#compaction#18 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:02,005 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 473b1abc37f649598dd6753d62fbafe6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1682693820619 2023-04-28 14:57:02,033 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): cce278287965607462f53e33830eba7b#B#compaction#20 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:02,079 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/ab2de3b398e044c1a460a7351930ff51 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/ab2de3b398e044c1a460a7351930ff51 2023-04-28 14:57:02,097 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/a446af92ed254fcf86edb27e2e59d64d as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/a446af92ed254fcf86edb27e2e59d64d 2023-04-28 14:57:02,100 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cce278287965607462f53e33830eba7b/A of cce278287965607462f53e33830eba7b into ab2de3b398e044c1a460a7351930ff51(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:02,103 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:02,103 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b., storeName=cce278287965607462f53e33830eba7b/A, priority=13, startTime=1682693821968; duration=0sec 2023-04-28 14:57:02,103 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:57:02,103 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:57:02,108 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35911 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:57:02,109 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): cce278287965607462f53e33830eba7b/C is initiating minor compaction (all files) 2023-04-28 14:57:02,109 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cce278287965607462f53e33830eba7b/C in TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:02,109 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/8f55aa855e0949a88d22561c2a4bc65b, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/2e44bde400f04fa281c4d11d0e9c90b9, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/61579ac381414e929dfca6576e6704ae] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp, totalSize=35.1 K 2023-04-28 14:57:02,114 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 8f55aa855e0949a88d22561c2a4bc65b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1682693819934 2023-04-28 14:57:02,116 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 2e44bde400f04fa281c4d11d0e9c90b9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1682693820535 2023-04-28 14:57:02,129 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 61579ac381414e929dfca6576e6704ae, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1682693820619 2023-04-28 14:57:02,130 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cce278287965607462f53e33830eba7b/B of cce278287965607462f53e33830eba7b into a446af92ed254fcf86edb27e2e59d64d(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:02,131 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:02,131 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b., storeName=cce278287965607462f53e33830eba7b/B, priority=13, startTime=1682693821969; duration=0sec 2023-04-28 14:57:02,132 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:02,146 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): cce278287965607462f53e33830eba7b#C#compaction#21 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:02,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on cce278287965607462f53e33830eba7b 2023-04-28 14:57:02,183 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:02,197 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/6b183ee10b03416886c54daacdd9f5a7 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/6b183ee10b03416886c54daacdd9f5a7 2023-04-28 14:57:02,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:02,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693882198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:02,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:02,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693882201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:02,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:02,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693882201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:02,211 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cce278287965607462f53e33830eba7b/C of cce278287965607462f53e33830eba7b into 6b183ee10b03416886c54daacdd9f5a7(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:02,213 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:02,213 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b., storeName=cce278287965607462f53e33830eba7b/C, priority=13, startTime=1682693821971; duration=0sec 2023-04-28 14:57:02,213 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:02,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:02,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693882207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:02,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:02,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693882209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:02,313 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:02,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693882311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:02,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:02,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:02,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693882312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:02,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693882312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:02,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:02,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693882317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:02,324 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:02,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693882319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:02,495 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/d592811967df4f9b9007ece20d8b86af 2023-04-28 14:57:02,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:02,546 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:02,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693882540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:02,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693882516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:02,548 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:02,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693882546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:02,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:02,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693882547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:02,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:02,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693882548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:02,554 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/af40aab89bdc4ad7843cb8f4d365aa53 2023-04-28 14:57:02,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:02,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693882849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:02,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:02,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693882854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:02,856 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:02,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693882854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:02,858 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:02,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693882856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:02,858 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:02,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693882857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:02,987 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/9a98a7e3198641b3bb8172ae10014ae3 2023-04-28 14:57:03,004 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/d592811967df4f9b9007ece20d8b86af as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/d592811967df4f9b9007ece20d8b86af 2023-04-28 14:57:03,012 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/d592811967df4f9b9007ece20d8b86af, entries=150, sequenceid=162, filesize=11.8 K 2023-04-28 14:57:03,014 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/af40aab89bdc4ad7843cb8f4d365aa53 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/af40aab89bdc4ad7843cb8f4d365aa53 2023-04-28 14:57:03,025 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/af40aab89bdc4ad7843cb8f4d365aa53, entries=150, sequenceid=162, filesize=11.8 K 2023-04-28 14:57:03,030 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/9a98a7e3198641b3bb8172ae10014ae3 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/9a98a7e3198641b3bb8172ae10014ae3 2023-04-28 14:57:03,039 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/9a98a7e3198641b3bb8172ae10014ae3, entries=150, sequenceid=162, filesize=11.8 K 2023-04-28 14:57:03,041 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for cce278287965607462f53e33830eba7b in 1049ms, sequenceid=162, compaction requested=false 2023-04-28 14:57:03,041 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:03,042 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:03,042 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:03,042 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:03,042 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,043 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:03,043 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:03,044 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:03,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:03,045 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:03,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:03,045 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:03,045 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:03,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:03,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:03,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:03,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:03,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8323af2[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:03,046 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:03,046 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:03,047 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:03,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:03,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:03,047 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,048 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:03,047 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:03,048 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:03,048 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:03,050 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:03,050 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,050 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:03,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,050 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:03,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:03,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:03,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:03,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:03,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:03,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:03,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:03,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:03,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:03,058 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:03,058 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:03,058 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:03,058 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:03,058 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:03,059 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:03,059 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:03,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:03,059 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:03,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:03,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:03,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:03,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:03,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:03,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:03,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:03,060 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:03,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:03,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:03,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:03,062 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:03,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:03,064 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,064 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:03,064 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:03,064 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,064 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:03,064 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:03,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:03,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:03,064 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:03,064 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:03,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:03,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:03,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:03,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:03,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:03,067 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:03,066 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:03,067 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:03,067 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:03,167 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:03,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:03,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:03,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:03,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:03,171 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:03,171 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:03,172 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:03,172 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:03,173 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:03,173 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:03,173 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,173 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:03,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:03,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:03,174 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,174 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:03,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:03,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:03,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:03,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:03,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:03,175 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:03,175 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:03,175 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:03,175 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:03,176 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:03,176 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:03,176 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:03,176 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:03,176 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:03,176 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cce278287965607462f53e33830eba7b 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-28 14:57:03,177 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=A 2023-04-28 14:57:03,177 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:03,177 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=B 2023-04-28 14:57:03,177 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:03,177 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=C 2023-04-28 14:57:03,177 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:03,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on cce278287965607462f53e33830eba7b 2023-04-28 14:57:03,358 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:03,377 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:03,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693883374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,377 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:03,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693883374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,378 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:03,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693883375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:03,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:03,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693883376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693883376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,481 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:03,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693883478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:03,482 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:03,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693883479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693883479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:03,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693883484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:03,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693883484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,598 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/fee7e46c741549b79fe1822b462797ce 2023-04-28 14:57:03,631 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/22f93f78dba547d6bdb54e7c711b12f8 2023-04-28 14:57:03,658 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/fcded43779504c198cba0fae83a96579 2023-04-28 14:57:03,668 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/fee7e46c741549b79fe1822b462797ce as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/fee7e46c741549b79fe1822b462797ce 2023-04-28 14:57:03,677 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/fee7e46c741549b79fe1822b462797ce, entries=150, sequenceid=180, filesize=11.8 K 2023-04-28 14:57:03,685 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/22f93f78dba547d6bdb54e7c711b12f8 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/22f93f78dba547d6bdb54e7c711b12f8 2023-04-28 14:57:03,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:03,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693883684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,690 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:03,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693883684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,690 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:03,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693883685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,691 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:03,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693883688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,692 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:03,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693883688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,698 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/22f93f78dba547d6bdb54e7c711b12f8, entries=150, sequenceid=180, filesize=11.8 K 2023-04-28 14:57:03,701 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/fcded43779504c198cba0fae83a96579 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/fcded43779504c198cba0fae83a96579 2023-04-28 14:57:03,711 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/fcded43779504c198cba0fae83a96579, entries=150, sequenceid=180, filesize=11.8 K 2023-04-28 14:57:03,713 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for cce278287965607462f53e33830eba7b in 537ms, sequenceid=180, compaction requested=true 2023-04-28 14:57:03,713 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:03,713 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:03,713 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:03,714 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:03,714 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,714 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:03,714 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:03,715 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,715 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:03,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:03,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:03,715 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:03,715 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:03,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:03,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:03,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:03,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:03,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:03,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d66c616[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:03,717 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:03,717 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:03,717 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:03,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:03,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:03,717 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:03,717 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,718 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:03,718 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:03,718 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:03,720 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:03,720 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,720 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:03,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:03,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:03,720 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:03,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:03,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:03,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:03,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:03,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:03,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:03,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:03,724 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:03,724 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:03,724 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:03,724 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:03,724 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:03,725 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:03,725 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:03,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:03,725 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:03,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:03,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:03,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:03,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:03,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:03,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:03,726 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:03,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:03,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:03,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:03,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:03,727 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:03,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:03,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,731 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,731 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:03,731 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:03,731 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:03,731 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:03,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:03,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:03,731 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:03,731 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:03,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:03,731 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:03,732 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:03,732 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:03,732 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:03,732 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:03,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:03,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:03,833 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:03,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:03,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:03,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:03,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:03,845 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:03,845 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:03,846 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:03,846 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:03,846 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:03,846 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:03,847 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:03,847 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:03,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:03,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:03,848 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:03,848 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:03,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:03,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:03,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:03,849 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:03,849 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:03,849 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:03,849 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:03,849 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:03,850 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:03,861 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:03,861 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:03,861 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:03,861 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:03,862 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cce278287965607462f53e33830eba7b 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-28 14:57:03,862 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=A 2023-04-28 14:57:03,862 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:03,862 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=B 2023-04-28 14:57:03,862 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:03,862 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=C 2023-04-28 14:57:03,862 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:03,891 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/32a28c1ecb7d4e4cbe536de007a3739c 2023-04-28 14:57:03,931 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/fad4a3db93b44a049708afcaf4c806e3 2023-04-28 14:57:03,959 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/c5a9755f97e640e69f4036fcad499e28 2023-04-28 14:57:03,968 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/32a28c1ecb7d4e4cbe536de007a3739c as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/32a28c1ecb7d4e4cbe536de007a3739c 2023-04-28 14:57:03,976 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/32a28c1ecb7d4e4cbe536de007a3739c, entries=150, sequenceid=198, filesize=11.8 K 2023-04-28 14:57:03,979 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/fad4a3db93b44a049708afcaf4c806e3 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/fad4a3db93b44a049708afcaf4c806e3 2023-04-28 14:57:03,987 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/fad4a3db93b44a049708afcaf4c806e3, entries=150, sequenceid=198, filesize=11.8 K 2023-04-28 14:57:03,991 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/c5a9755f97e640e69f4036fcad499e28 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/c5a9755f97e640e69f4036fcad499e28 2023-04-28 14:57:03,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on cce278287965607462f53e33830eba7b 2023-04-28 14:57:03,999 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:04,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:04,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693884006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,015 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:04,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693884008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,016 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:04,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693884010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:04,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693884012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:04,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693884012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,024 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/c5a9755f97e640e69f4036fcad499e28, entries=150, sequenceid=198, filesize=11.8 K 2023-04-28 14:57:04,026 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for cce278287965607462f53e33830eba7b in 164ms, sequenceid=198, compaction requested=true 2023-04-28 14:57:04,026 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:04,026 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:04,026 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:04,026 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:04,026 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,026 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:04,026 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:04,028 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,028 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:04,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:04,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:04,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:04,029 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:04,029 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:04,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:04,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:04,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:04,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:04,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f8605[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:04,031 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:04,031 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:04,034 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:04,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:04,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:04,035 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:04,035 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:04,035 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:04,035 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,035 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:04,036 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,036 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:04,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:04,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:04,036 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:04,036 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:04,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:04,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:04,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:04,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:04,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:04,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:04,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:04,042 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:04,042 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:04,042 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:04,042 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:04,042 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:04,044 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:04,044 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:04,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:04,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:04,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:04,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:04,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:04,044 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:04,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:04,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:04,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:04,045 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:04,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:04,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:04,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:04,046 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:04,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:04,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,050 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:04,050 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,050 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:04,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:04,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:04,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:04,050 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:04,052 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:04,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:04,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:04,051 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:04,052 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,052 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:04,052 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:04,052 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:04,052 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:04,052 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:04,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:04,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:04,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on cce278287965607462f53e33830eba7b 2023-04-28 14:57:04,134 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cce278287965607462f53e33830eba7b 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-28 14:57:04,135 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=A 2023-04-28 14:57:04,135 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:04,135 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=B 2023-04-28 14:57:04,139 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:04,139 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=C 2023-04-28 14:57:04,139 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:04,153 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:04,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:04,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:04,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:04,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:04,184 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:04,184 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:04,191 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:04,191 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:04,191 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:04,191 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:04,192 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:04,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:04,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:04,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:04,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:04,193 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:04,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:04,194 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:04,194 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,194 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:04,197 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:04,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693884173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,198 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:04,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693884175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,198 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:04,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693884179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,199 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:04,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693884184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,199 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:04,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693884185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,213 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/c6e03d9edda04c6ca8d23f5fd63c1315 2023-04-28 14:57:04,230 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:04,230 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:04,230 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:04,247 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:04,249 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:04,249 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:04,249 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:04,249 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:04,250 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:04,250 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:04,250 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:04,250 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:04,250 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,250 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:04,250 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:04,251 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:04,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:04,252 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:04,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:04,252 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:04,252 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:04,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:04,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:04,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:04,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:04,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64170103[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:04,254 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:04,254 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:04,256 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:04,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:04,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:04,256 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:04,256 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:04,256 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:04,257 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,257 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:04,258 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,258 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:04,258 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:04,258 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:04,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:04,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:04,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:04,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:04,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:04,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:04,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:04,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:04,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:04,262 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:04,262 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:04,262 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:04,262 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:04,262 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:04,269 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:04,270 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:04,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:04,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:04,270 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:04,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:04,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:04,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:04,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:04,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:04,271 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:04,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:04,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:04,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:04,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:04,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,276 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:04,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:04,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,278 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:04,278 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:04,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:04,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:04,279 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,279 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:04,279 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:04,279 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:04,279 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:04,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:04,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:04,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:04,282 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:04,282 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:04,279 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:04,282 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:04,282 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:04,299 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/80363376c29448ce96cd9c00c6382999 2023-04-28 14:57:04,301 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:04,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693884299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,302 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:04,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693884299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:04,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693884300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,303 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:04,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693884301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:04,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693884301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,341 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/f1192348fc884b2283ab04f539b31400 2023-04-28 14:57:04,354 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/c6e03d9edda04c6ca8d23f5fd63c1315 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/c6e03d9edda04c6ca8d23f5fd63c1315 2023-04-28 14:57:04,368 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/c6e03d9edda04c6ca8d23f5fd63c1315, entries=150, sequenceid=213, filesize=11.8 K 2023-04-28 14:57:04,371 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/80363376c29448ce96cd9c00c6382999 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/80363376c29448ce96cd9c00c6382999 2023-04-28 14:57:04,381 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/80363376c29448ce96cd9c00c6382999, entries=150, sequenceid=213, filesize=11.8 K 2023-04-28 14:57:04,386 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:04,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:04,388 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/f1192348fc884b2283ab04f539b31400 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/f1192348fc884b2283ab04f539b31400 2023-04-28 14:57:04,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:04,398 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/f1192348fc884b2283ab04f539b31400, entries=150, sequenceid=213, filesize=11.8 K 2023-04-28 14:57:04,399 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for cce278287965607462f53e33830eba7b in 265ms, sequenceid=213, compaction requested=true 2023-04-28 14:57:04,399 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:04,399 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 14:57:04,400 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:04,400 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 14:57:04,400 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:04,400 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:57:04,402 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60386 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 14:57:04,402 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): cce278287965607462f53e33830eba7b/A is initiating minor compaction (all files) 2023-04-28 14:57:04,402 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cce278287965607462f53e33830eba7b/A in TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:04,402 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/ab2de3b398e044c1a460a7351930ff51, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/d592811967df4f9b9007ece20d8b86af, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/fee7e46c741549b79fe1822b462797ce, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/32a28c1ecb7d4e4cbe536de007a3739c, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/c6e03d9edda04c6ca8d23f5fd63c1315] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp, totalSize=59.0 K 2023-04-28 14:57:04,403 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting ab2de3b398e044c1a460a7351930ff51, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1682693820619 2023-04-28 14:57:04,403 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60386 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 14:57:04,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:04,406 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): cce278287965607462f53e33830eba7b/B is initiating minor compaction (all files) 2023-04-28 14:57:04,403 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting d592811967df4f9b9007ece20d8b86af, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1682693821003 2023-04-28 14:57:04,406 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cce278287965607462f53e33830eba7b/B in TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:04,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:04,406 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:04,407 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/a446af92ed254fcf86edb27e2e59d64d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/af40aab89bdc4ad7843cb8f4d365aa53, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/22f93f78dba547d6bdb54e7c711b12f8, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/fad4a3db93b44a049708afcaf4c806e3, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/80363376c29448ce96cd9c00c6382999] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp, totalSize=59.0 K 2023-04-28 14:57:04,407 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:04,408 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:04,408 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:04,409 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:04,409 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting a446af92ed254fcf86edb27e2e59d64d, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1682693820619 2023-04-28 14:57:04,409 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:04,409 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting af40aab89bdc4ad7843cb8f4d365aa53, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1682693821003 2023-04-28 14:57:04,410 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:04,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:04,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:04,410 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:04,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:04,411 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 22f93f78dba547d6bdb54e7c711b12f8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1682693822192 2023-04-28 14:57:04,411 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:04,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:04,411 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:04,411 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:04,412 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting fad4a3db93b44a049708afcaf4c806e3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1682693823372 2023-04-28 14:57:04,412 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 80363376c29448ce96cd9c00c6382999, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682693824009 2023-04-28 14:57:04,414 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting fee7e46c741549b79fe1822b462797ce, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1682693822192 2023-04-28 14:57:04,415 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 32a28c1ecb7d4e4cbe536de007a3739c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1682693823372 2023-04-28 14:57:04,415 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting c6e03d9edda04c6ca8d23f5fd63c1315, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682693824009 2023-04-28 14:57:04,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:04,420 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:04,420 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:04,420 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:04,422 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:04,423 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:04,423 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:04,423 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:04,424 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:04,424 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cce278287965607462f53e33830eba7b 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-28 14:57:04,424 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=A 2023-04-28 14:57:04,424 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:04,424 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=B 2023-04-28 14:57:04,425 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:04,425 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=C 2023-04-28 14:57:04,425 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:04,456 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): cce278287965607462f53e33830eba7b#A#compaction#34 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:04,467 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): cce278287965607462f53e33830eba7b#B#compaction#35 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:04,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on cce278287965607462f53e33830eba7b 2023-04-28 14:57:04,525 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:04,531 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/7b03223476a24ded87e6f85f1183b80c 2023-04-28 14:57:04,560 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:04,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693884548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:04,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693884548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:04,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:04,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693884554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693884560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,573 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:04,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693884562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:04,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693884662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:04,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693884664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:04,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693884674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:04,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:04,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693884674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693884675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:04,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693884866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:04,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693884866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:04,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693884878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:04,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693884879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,883 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:04,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693884879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:04,920 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/96b260db18aa4cfb8c1c311511caaff7 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/96b260db18aa4cfb8c1c311511caaff7 2023-04-28 14:57:04,922 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/03d005ce43414181aced3fb39fe09228 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/03d005ce43414181aced3fb39fe09228 2023-04-28 14:57:04,932 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in cce278287965607462f53e33830eba7b/A of cce278287965607462f53e33830eba7b into 03d005ce43414181aced3fb39fe09228(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:04,932 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in cce278287965607462f53e33830eba7b/B of cce278287965607462f53e33830eba7b into 96b260db18aa4cfb8c1c311511caaff7(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:04,933 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:04,933 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:04,933 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b., storeName=cce278287965607462f53e33830eba7b/A, priority=11, startTime=1682693824399; duration=0sec 2023-04-28 14:57:04,933 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b., storeName=cce278287965607462f53e33830eba7b/B, priority=11, startTime=1682693824400; duration=0sec 2023-04-28 14:57:04,933 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:57:04,933 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:04,933 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 14:57:04,939 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60386 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 14:57:04,939 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): cce278287965607462f53e33830eba7b/C is initiating minor compaction (all files) 2023-04-28 14:57:04,939 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cce278287965607462f53e33830eba7b/C in TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:04,939 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/6b183ee10b03416886c54daacdd9f5a7, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/9a98a7e3198641b3bb8172ae10014ae3, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/fcded43779504c198cba0fae83a96579, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/c5a9755f97e640e69f4036fcad499e28, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/f1192348fc884b2283ab04f539b31400] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp, totalSize=59.0 K 2023-04-28 14:57:04,940 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 6b183ee10b03416886c54daacdd9f5a7, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1682693820619 2023-04-28 14:57:04,940 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 9a98a7e3198641b3bb8172ae10014ae3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1682693821003 2023-04-28 14:57:04,941 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting fcded43779504c198cba0fae83a96579, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1682693822192 2023-04-28 14:57:04,941 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting c5a9755f97e640e69f4036fcad499e28, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1682693823372 2023-04-28 14:57:04,941 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting f1192348fc884b2283ab04f539b31400, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682693824009 2023-04-28 14:57:04,957 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): cce278287965607462f53e33830eba7b#C#compaction#37 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:04,998 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/1d252876fadb4e09aff9b1f8ace7cf56 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/1d252876fadb4e09aff9b1f8ace7cf56 2023-04-28 14:57:05,001 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/efef38f3ded34c37aabf653b8bd2714f 2023-04-28 14:57:05,013 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in cce278287965607462f53e33830eba7b/C of cce278287965607462f53e33830eba7b into 1d252876fadb4e09aff9b1f8ace7cf56(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:05,013 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:05,014 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b., storeName=cce278287965607462f53e33830eba7b/C, priority=11, startTime=1682693824400; duration=0sec 2023-04-28 14:57:05,014 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:05,033 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/956994df71a0433bb625328da3dd5142 2023-04-28 14:57:05,047 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/7b03223476a24ded87e6f85f1183b80c as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/7b03223476a24ded87e6f85f1183b80c 2023-04-28 14:57:05,057 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/7b03223476a24ded87e6f85f1183b80c, entries=150, sequenceid=231, filesize=11.8 K 2023-04-28 14:57:05,058 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/efef38f3ded34c37aabf653b8bd2714f as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/efef38f3ded34c37aabf653b8bd2714f 2023-04-28 14:57:05,071 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/efef38f3ded34c37aabf653b8bd2714f, entries=150, sequenceid=231, filesize=11.8 K 2023-04-28 14:57:05,073 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/956994df71a0433bb625328da3dd5142 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/956994df71a0433bb625328da3dd5142 2023-04-28 14:57:05,081 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/956994df71a0433bb625328da3dd5142, entries=150, sequenceid=231, filesize=11.8 K 2023-04-28 14:57:05,082 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for cce278287965607462f53e33830eba7b in 658ms, sequenceid=231, compaction requested=false 2023-04-28 14:57:05,082 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:05,082 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:05,083 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:05,083 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:05,083 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,086 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:05,086 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:05,087 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,087 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:05,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:05,088 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,088 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:05,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:05,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:05,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:05,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:05,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7deb52d9[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:05,090 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:05,090 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,091 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,091 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:05,091 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:05,091 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:05,094 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:05,094 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,094 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:05,094 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:05,099 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:05,099 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:05,099 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:05,099 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:05,099 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:05,099 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:05,099 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:05,101 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,101 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:05,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,101 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:05,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:05,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:05,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:05,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:05,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,103 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:05,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,104 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:05,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,107 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,107 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,107 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,107 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,107 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,107 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:05,107 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:05,107 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:05,111 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,111 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:05,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:05,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:05,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:05,111 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:05,111 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:05,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:05,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:05,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on cce278287965607462f53e33830eba7b 2023-04-28 14:57:05,172 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cce278287965607462f53e33830eba7b 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-28 14:57:05,174 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=A 2023-04-28 14:57:05,174 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:05,174 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=B 2023-04-28 14:57:05,174 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:05,174 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=C 2023-04-28 14:57:05,174 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:05,188 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/e5302fba6efa4bdc9e8e2d4f9d4d4a0a 2023-04-28 14:57:05,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:05,200 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:05,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693885195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693885195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,200 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:05,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693885195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,203 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:05,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693885200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,204 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:05,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693885200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,211 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:05,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:05,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:05,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:05,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:05,215 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:05,215 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:05,216 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:05,216 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:05,217 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,217 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,218 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,218 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:05,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:05,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:05,218 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,218 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:05,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:05,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,219 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:05,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:05,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:05,222 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:05,222 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:05,223 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:05,226 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:05,226 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:05,226 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:05,226 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:05,226 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:05,226 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:05,226 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:05,226 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:05,226 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,226 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:05,226 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:05,228 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,228 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,228 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:05,230 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:05,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:05,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:05,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:05,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:05,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:05,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16566d04[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:05,232 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:05,232 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,232 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,233 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,233 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:05,233 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:05,233 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:05,233 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:05,234 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:05,234 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:05,234 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:05,235 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:05,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:05,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:05,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:05,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:05,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:05,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:05,237 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:05,237 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:05,237 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:05,237 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:05,237 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:05,238 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,238 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,238 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:05,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:05,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:05,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:05,239 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:05,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:05,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:05,240 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:05,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,243 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:05,243 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,243 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:05,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:05,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:05,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:05,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:05,243 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,243 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,243 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,243 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,243 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,243 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,243 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:05,243 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:05,244 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:05,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:05,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:05,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:05,304 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:05,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693885302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,304 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:05,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693885302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,305 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:05,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693885303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,307 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:05,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693885306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,309 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:05,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693885306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,344 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:05,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:05,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:05,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:05,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:05,350 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:05,350 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:05,351 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:05,351 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:05,352 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,352 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,352 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:05,352 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:05,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:05,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,353 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,353 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:05,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:05,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:05,354 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:05,354 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:05,354 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:05,354 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:05,355 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:05,358 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:05,358 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:05,358 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:05,358 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:05,358 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:05,358 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:05,358 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:05,358 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:05,358 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,358 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:05,358 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:05,360 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,360 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:05,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:05,360 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,360 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:05,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:05,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:05,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:05,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:05,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a759e65[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:05,363 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:05,363 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,365 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,365 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,365 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:05,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,365 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:05,365 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:05,365 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:05,368 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:05,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:05,368 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:05,368 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:05,368 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:05,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:05,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:05,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:05,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:05,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:05,372 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:05,372 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:05,373 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:05,373 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:05,373 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:05,374 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,374 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,374 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:05,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:05,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:05,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:05,375 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:05,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:05,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:05,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,376 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:05,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,378 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:05,378 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,378 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,378 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,378 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:05,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:05,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:05,378 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:05,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:05,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:05,378 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,379 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,379 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,379 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,378 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:05,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:05,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:05,379 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:05,379 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:05,480 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:05,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:05,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:05,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:05,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:05,494 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:05,494 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:05,495 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:05,495 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:05,496 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,496 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,497 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:05,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:05,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:05,497 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,498 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,498 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:05,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:05,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,499 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:05,502 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:05,502 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:05,502 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:05,502 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:05,503 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:05,506 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:05,506 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:05,506 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:05,506 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:05,506 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:05,506 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:05,506 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:05,506 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:05,506 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,506 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:05,506 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:05,508 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,508 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:05,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:05,508 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:05,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693885507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,508 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,508 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:05,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:05,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:05,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:05,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693885509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693885509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:05,514 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:05,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693885511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:05,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:05,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693885511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:05,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36369531[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:05,518 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:05,518 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,519 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,519 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:05,519 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:05,519 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:05,520 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,520 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:05,521 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:05,521 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,521 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:05,521 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:05,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:05,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:05,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:05,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:05,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:05,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:05,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:05,525 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:05,525 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:05,525 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:05,525 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:05,525 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:05,527 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,527 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:05,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,527 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:05,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:05,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:05,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:05,528 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:05,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:05,538 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:05,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,541 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:05,541 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:05,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:05,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:05,541 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,541 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,541 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,541 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,541 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,542 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,542 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,542 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:05,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:05,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:05,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:05,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:05,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:05,558 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:05,558 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:05,610 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/a790f9dc2ff04f20ad2ec4a699d34f1a 2023-04-28 14:57:05,658 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:05,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:05,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:05,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:05,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:05,664 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:05,664 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:05,665 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:05,665 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:05,669 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,669 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,670 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:05,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:05,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:05,670 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,670 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,670 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:05,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:05,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,672 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:05,678 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:05,678 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:05,678 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:05,678 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:05,685 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:05,685 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:05,685 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:05,685 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:05,685 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:05,685 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:05,685 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:05,686 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:05,686 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:05,686 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,686 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:05,686 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:05,687 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,688 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,688 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:05,688 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:05,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:05,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:05,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:05,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:05,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:05,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11ec74b4[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:05,701 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:05,701 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,702 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,702 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:05,702 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:05,702 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:05,703 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,703 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:05,704 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:05,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:05,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:05,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:05,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:05,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:05,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:05,707 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:05,707 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:05,707 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:05,707 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:05,707 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:05,708 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:05,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:05,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:05,709 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,710 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,710 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:05,710 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:05,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:05,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,711 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:05,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,713 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,713 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,714 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,714 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,714 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,714 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,714 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,710 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:05,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:05,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:05,713 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:05,714 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:05,721 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:05,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:05,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:05,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:05,726 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:05,726 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:05,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:05,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:05,813 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:05,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693885811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:05,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693885811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:05,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693885814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:05,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693885819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:05,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693885820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,827 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:05,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:05,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:05,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:05,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:05,830 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:05,830 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:05,831 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:05,831 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:05,831 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,831 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,832 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:05,832 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:05,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:05,832 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,832 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:05,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:05,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,833 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:05,833 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:05,833 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:05,833 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:05,833 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:05,833 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:05,833 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:05,833 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:05,833 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:05,833 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:05,834 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:05,834 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:05,834 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:05,834 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:05,834 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,834 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:05,834 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:05,836 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,836 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:05,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:05,836 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:05,836 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:05,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:05,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:05,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:05,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@347715c0[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:05,837 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:05,837 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,838 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,838 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:05,838 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:05,838 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:05,838 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,838 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:05,839 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:05,839 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,839 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:05,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:05,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:05,839 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:05,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:05,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:05,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:05,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:05,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:05,842 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:05,842 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:05,842 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:05,842 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:05,842 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:05,843 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:05,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:05,843 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,843 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:05,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:05,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:05,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:05,844 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:05,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,845 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:05,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,848 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:05,848 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:05,848 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:05,848 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,848 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,848 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,848 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,848 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,848 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,848 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:05,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:05,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:05,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:05,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:05,849 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:05,849 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:05,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:05,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:05,949 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:05,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:05,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:05,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:05,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:05,957 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:05,957 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:05,957 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:05,957 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:05,958 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,958 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,959 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:05,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:05,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:05,959 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,960 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,960 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:05,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:05,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,961 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:05,961 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:05,961 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:05,962 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:05,962 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:05,962 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:05,962 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:05,962 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:05,962 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:05,962 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:05,962 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:05,962 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:05,962 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:05,962 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:05,962 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,962 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:05,962 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:05,963 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,963 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:05,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:05,964 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,964 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:05,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:05,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:05,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:05,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:05,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7237239d[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:05,965 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:05,965 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,966 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,966 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:05,966 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:05,966 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:05,967 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,967 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:05,967 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,967 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:05,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:05,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:05,967 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:05,968 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:05,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:05,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:05,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:05,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:05,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:05,969 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:05,970 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:05,970 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:05,970 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:05,970 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:05,971 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,971 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:05,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,971 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:05,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:05,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:05,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:05,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:05,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,972 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:05,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:05,974 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:05,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:05,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:05,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:05,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:05,977 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:05,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:05,977 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:05,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:05,977 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:05,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:05,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:05,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:05,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:05,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:05,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:06,065 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/088f581f7ce84f5a90640eac49332747 2023-04-28 14:57:06,071 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/e5302fba6efa4bdc9e8e2d4f9d4d4a0a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/e5302fba6efa4bdc9e8e2d4f9d4d4a0a 2023-04-28 14:57:06,077 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:06,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:06,079 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/e5302fba6efa4bdc9e8e2d4f9d4d4a0a, entries=150, sequenceid=251, filesize=11.8 K 2023-04-28 14:57:06,080 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/a790f9dc2ff04f20ad2ec4a699d34f1a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/a790f9dc2ff04f20ad2ec4a699d34f1a 2023-04-28 14:57:06,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:06,089 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/a790f9dc2ff04f20ad2ec4a699d34f1a, entries=150, sequenceid=251, filesize=11.8 K 2023-04-28 14:57:06,090 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/088f581f7ce84f5a90640eac49332747 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/088f581f7ce84f5a90640eac49332747 2023-04-28 14:57:06,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:06,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:06,098 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:06,098 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:06,098 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:06,098 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:06,098 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:06,098 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:06,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:06,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:06,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:06,099 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:06,100 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,100 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:06,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:06,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:06,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:06,100 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:06,101 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:06,101 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:06,101 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:06,101 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:06,101 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:06,101 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:06,101 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:06,102 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:06,102 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:06,102 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:06,102 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:06,102 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:06,102 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:06,102 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,102 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:06,102 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:06,103 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,103 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:06,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:06,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:06,104 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:06,104 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:06,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:06,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:06,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:06,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:06,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:06,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@702abff7[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:06,105 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:06,105 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:06,106 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:06,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:06,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:06,106 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:06,106 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,106 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:06,106 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:06,106 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:06,107 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/088f581f7ce84f5a90640eac49332747, entries=150, sequenceid=251, filesize=11.8 K 2023-04-28 14:57:06,107 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,107 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:06,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:06,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:06,108 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:06,108 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:06,108 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for cce278287965607462f53e33830eba7b in 936ms, sequenceid=251, compaction requested=true 2023-04-28 14:57:06,108 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:06,109 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:06,109 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:57:06,109 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:57:06,109 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:57:06,110 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:57:06,110 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:57:06,112 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): cce278287965607462f53e33830eba7b/B is initiating minor compaction (all files) 2023-04-28 14:57:06,112 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cce278287965607462f53e33830eba7b/B in TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:06,112 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/96b260db18aa4cfb8c1c311511caaff7, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/efef38f3ded34c37aabf653b8bd2714f, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/a790f9dc2ff04f20ad2ec4a699d34f1a] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp, totalSize=35.6 K 2023-04-28 14:57:06,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:06,112 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:57:06,112 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): cce278287965607462f53e33830eba7b/A is initiating minor compaction (all files) 2023-04-28 14:57:06,113 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cce278287965607462f53e33830eba7b/A in TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:06,113 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/03d005ce43414181aced3fb39fe09228, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/7b03223476a24ded87e6f85f1183b80c, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/e5302fba6efa4bdc9e8e2d4f9d4d4a0a] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp, totalSize=35.6 K 2023-04-28 14:57:06,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:06,113 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 03d005ce43414181aced3fb39fe09228, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682693824009 2023-04-28 14:57:06,113 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 96b260db18aa4cfb8c1c311511caaff7, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682693824009 2023-04-28 14:57:06,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:06,114 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 7b03223476a24ded87e6f85f1183b80c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1682693824171 2023-04-28 14:57:06,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,114 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting efef38f3ded34c37aabf653b8bd2714f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1682693824171 2023-04-28 14:57:06,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:06,115 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting a790f9dc2ff04f20ad2ec4a699d34f1a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1682693824557 2023-04-28 14:57:06,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:06,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,116 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting e5302fba6efa4bdc9e8e2d4f9d4d4a0a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1682693824557 2023-04-28 14:57:06,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:06,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:06,116 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:06,116 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:06,116 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:06,116 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:06,116 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:06,120 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:06,121 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:06,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:06,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:06,121 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:06,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:06,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:06,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:06,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:06,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:06,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:06,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:06,122 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:06,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:06,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:06,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,125 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:06,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:06,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,130 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,130 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:06,130 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:06,130 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:06,130 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:06,130 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:06,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:06,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:06,130 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:06,130 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:06,130 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:06,130 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:06,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:06,130 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:06,131 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:06,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:06,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:06,147 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): cce278287965607462f53e33830eba7b#B#compaction#42 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:06,150 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): cce278287965607462f53e33830eba7b#A#compaction#43 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:06,205 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/cb01a112c018439e90757ec6069a94ad as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/cb01a112c018439e90757ec6069a94ad 2023-04-28 14:57:06,215 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cce278287965607462f53e33830eba7b/B of cce278287965607462f53e33830eba7b into cb01a112c018439e90757ec6069a94ad(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:06,215 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:06,215 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b., storeName=cce278287965607462f53e33830eba7b/B, priority=13, startTime=1682693826109; duration=0sec 2023-04-28 14:57:06,215 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:57:06,215 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:57:06,217 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:57:06,217 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): cce278287965607462f53e33830eba7b/C is initiating minor compaction (all files) 2023-04-28 14:57:06,217 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cce278287965607462f53e33830eba7b/C in TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:06,217 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/1d252876fadb4e09aff9b1f8ace7cf56, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/956994df71a0433bb625328da3dd5142, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/088f581f7ce84f5a90640eac49332747] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp, totalSize=35.6 K 2023-04-28 14:57:06,217 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 1d252876fadb4e09aff9b1f8ace7cf56, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682693824009 2023-04-28 14:57:06,218 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 956994df71a0433bb625328da3dd5142, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1682693824171 2023-04-28 14:57:06,218 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 088f581f7ce84f5a90640eac49332747, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1682693824557 2023-04-28 14:57:06,225 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): cce278287965607462f53e33830eba7b#C#compaction#44 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:06,231 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:06,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:06,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:06,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:06,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:06,238 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:06,238 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:06,238 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:06,238 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:06,240 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:06,240 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:06,241 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:06,241 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:06,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:06,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:06,241 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,241 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:06,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:06,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:06,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:06,242 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:06,262 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:06,262 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:06,262 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:06,262 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:06,262 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:06,266 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:06,266 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:06,266 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:06,266 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:06,266 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cce278287965607462f53e33830eba7b 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-28 14:57:06,267 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=A 2023-04-28 14:57:06,267 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:06,267 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=B 2023-04-28 14:57:06,267 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:06,267 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=C 2023-04-28 14:57:06,267 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:06,284 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/83455c337ff4444bb4b2764de10b95fb 2023-04-28 14:57:06,318 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:06,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on cce278287965607462f53e33830eba7b 2023-04-28 14:57:06,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:06,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693886338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:06,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693886339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:06,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693886340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,343 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:06,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693886341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,345 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:06,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693886341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:06,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693886445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:06,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693886446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:06,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693886447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,450 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:06,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693886448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:06,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693886448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,603 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/27f76f301fbc4f52adcf1ac9dc93cdb2 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/27f76f301fbc4f52adcf1ac9dc93cdb2 2023-04-28 14:57:06,615 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cce278287965607462f53e33830eba7b/A of cce278287965607462f53e33830eba7b into 27f76f301fbc4f52adcf1ac9dc93cdb2(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:06,615 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:06,615 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b., storeName=cce278287965607462f53e33830eba7b/A, priority=13, startTime=1682693826109; duration=0sec 2023-04-28 14:57:06,616 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:06,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:06,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693886650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:06,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693886650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:06,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693886650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:06,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693886651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:06,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53108 deadline: 1682693886655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,670 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/18c8c5124a7b436e8214367177e7761e as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/18c8c5124a7b436e8214367177e7761e 2023-04-28 14:57:06,683 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cce278287965607462f53e33830eba7b/C of cce278287965607462f53e33830eba7b into 18c8c5124a7b436e8214367177e7761e(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:06,683 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:06,683 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b., storeName=cce278287965607462f53e33830eba7b/C, priority=13, startTime=1682693826110; duration=0sec 2023-04-28 14:57:06,683 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:06,709 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/f498f8c613cb4ce1810a37526b69b17a 2023-04-28 14:57:06,737 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/ba43580190f5438b98af185d3ef061c5 2023-04-28 14:57:06,749 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/83455c337ff4444bb4b2764de10b95fb as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/83455c337ff4444bb4b2764de10b95fb 2023-04-28 14:57:06,759 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/83455c337ff4444bb4b2764de10b95fb, entries=150, sequenceid=268, filesize=11.9 K 2023-04-28 14:57:06,760 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/f498f8c613cb4ce1810a37526b69b17a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/f498f8c613cb4ce1810a37526b69b17a 2023-04-28 14:57:06,766 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/f498f8c613cb4ce1810a37526b69b17a, entries=150, sequenceid=268, filesize=11.9 K 2023-04-28 14:57:06,768 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/ba43580190f5438b98af185d3ef061c5 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/ba43580190f5438b98af185d3ef061c5 2023-04-28 14:57:06,774 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/ba43580190f5438b98af185d3ef061c5, entries=150, sequenceid=268, filesize=11.9 K 2023-04-28 14:57:06,775 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for cce278287965607462f53e33830eba7b in 509ms, sequenceid=268, compaction requested=false 2023-04-28 14:57:06,775 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:06,775 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:06,775 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:06,775 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:06,775 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,776 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:06,776 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:06,777 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,777 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:06,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:06,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:06,778 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:06,778 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:06,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:06,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:06,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:06,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:06,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:06,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2912c92[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:06,784 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:06,784 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:06,784 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:06,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:06,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:06,785 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:06,785 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:06,785 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:06,785 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,785 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:06,786 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:06,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:06,791 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:06,792 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:06,792 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:06,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:06,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:06,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:06,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:06,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:06,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:06,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:06,796 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:06,796 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:06,796 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:06,796 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:06,796 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:06,798 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:06,798 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:06,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:06,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:06,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:06,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:06,798 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:06,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:06,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:06,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:06,798 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:06,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:06,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:06,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:06,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:06,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,800 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:06,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:06,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,804 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:06,804 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:06,804 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,804 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:06,804 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:06,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:06,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:06,804 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,804 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:06,804 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:06,804 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:06,805 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:06,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:06,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:06,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:06,806 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:06,806 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:06,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:06,906 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:06,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:06,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:06,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:06,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:06,922 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:06,922 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:06,922 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:06,922 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:06,923 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:06,923 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:06,923 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,923 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:06,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:06,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:06,924 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,924 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:06,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:06,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:06,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:06,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:06,925 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:06,926 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:06,926 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:06,926 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:06,926 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:06,926 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:06,930 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:06,930 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:06,932 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:06,932 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:06,933 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cce278287965607462f53e33830eba7b 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-28 14:57:06,933 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=A 2023-04-28 14:57:06,933 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:06,933 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=B 2023-04-28 14:57:06,933 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:06,933 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=C 2023-04-28 14:57:06,933 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:06,958 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:06,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on cce278287965607462f53e33830eba7b 2023-04-28 14:57:06,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:06,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693886968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,979 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:06,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:06,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693886973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693886974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,979 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:06,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693886975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:06,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:06,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693886974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:07,079 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:07,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693887076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:07,081 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:07,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693887081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:07,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:07,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693887081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:07,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:07,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693887081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:07,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:07,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693887086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:07,289 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:07,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693887288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:07,290 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:07,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693887288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:07,290 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:07,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693887287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:07,300 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:07,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693887292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:07,300 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:07,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693887293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:07,373 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/92e9978209b4410bb869cde95a40ce9e 2023-04-28 14:57:07,595 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:07,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:07,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693887593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:07,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693887595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:07,596 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:07,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693887595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:07,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:07,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693887602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:07,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:07,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693887603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:07,835 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/b708905bbf4849a6a39bfae2827e9a8b 2023-04-28 14:57:07,903 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/737e68e8f0cf40cf8ba35195a357f97c 2023-04-28 14:57:07,920 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/92e9978209b4410bb869cde95a40ce9e as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/92e9978209b4410bb869cde95a40ce9e 2023-04-28 14:57:07,927 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/92e9978209b4410bb869cde95a40ce9e, entries=150, sequenceid=287, filesize=11.9 K 2023-04-28 14:57:07,928 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/b708905bbf4849a6a39bfae2827e9a8b as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/b708905bbf4849a6a39bfae2827e9a8b 2023-04-28 14:57:07,939 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/b708905bbf4849a6a39bfae2827e9a8b, entries=150, sequenceid=287, filesize=11.9 K 2023-04-28 14:57:07,940 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/737e68e8f0cf40cf8ba35195a357f97c as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/737e68e8f0cf40cf8ba35195a357f97c 2023-04-28 14:57:07,948 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/737e68e8f0cf40cf8ba35195a357f97c, entries=150, sequenceid=287, filesize=11.9 K 2023-04-28 14:57:07,949 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for cce278287965607462f53e33830eba7b in 1017ms, sequenceid=287, compaction requested=true 2023-04-28 14:57:07,949 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:07,949 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:07,949 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:07,949 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:07,949 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:07,949 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:07,949 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:07,951 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:07,951 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:07,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:07,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:07,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:07,951 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:07,951 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:07,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:07,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:07,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:07,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:07,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:07,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:07,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ec4f015[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:07,953 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:07,953 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:07,954 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:07,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:07,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:07,954 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:07,954 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:07,954 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:07,954 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:07,954 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:07,955 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:07,955 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:07,955 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:07,955 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:07,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:07,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:07,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:07,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:07,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:07,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:07,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:07,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:07,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:07,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:07,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:07,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:07,959 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:07,959 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:07,959 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:07,959 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:07,959 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:07,961 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:07,961 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:07,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:07,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:07,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:07,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:07,961 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:07,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:07,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:07,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:07,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:07,962 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:07,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:07,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:07,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:07,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:07,964 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:07,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:07,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:07,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:07,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:07,965 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:07,965 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:07,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:07,966 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:07,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:07,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:07,966 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:07,966 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:07,966 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:07,966 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:07,966 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:07,966 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:07,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:07,966 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:07,966 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:07,966 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:07,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:07,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:07,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:08,066 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:08,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:08,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:08,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:08,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:08,077 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:08,077 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:08,077 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:08,077 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:08,078 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:08,078 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:08,079 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:08,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:08,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:08,079 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:08,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:08,079 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:08,079 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:08,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:08,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:08,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:08,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:08,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:08,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:08,080 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:08,080 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:08,081 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:08,085 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:08,085 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:08,085 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:08,087 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:08,088 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cce278287965607462f53e33830eba7b 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-28 14:57:08,090 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=A 2023-04-28 14:57:08,090 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:08,090 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=B 2023-04-28 14:57:08,090 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:08,090 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=C 2023-04-28 14:57:08,090 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:08,101 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/df9fb7c8121c498e89c5891ea713f5bd 2023-04-28 14:57:08,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on cce278287965607462f53e33830eba7b 2023-04-28 14:57:08,107 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:08,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:08,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693888126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:08,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:08,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693888128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:08,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:08,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693888132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:08,138 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:08,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53134 deadline: 1682693888133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:08,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:08,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53138 deadline: 1682693888135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:08,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:08,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693888236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:08,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:08,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693888239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:08,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:08,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693888239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:08,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:08,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693888241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:08,243 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:08,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693888241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:08,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:08,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693888442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:08,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:08,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53136 deadline: 1682693888444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:08,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:08,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693888444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:08,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:08,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693888445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:08,447 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:08,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693888446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:08,553 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/a7396c0b9cc946efb904be055e45f5bf 2023-04-28 14:57:08,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:08,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693888748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:08,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:08,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693888748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:08,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:08,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693888749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:08,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:08,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693888750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:08,754 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:08,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:53114 deadline: 1682693888752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,001 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/9002a2ab62644ca08949a617d26ea42b 2023-04-28 14:57:09,009 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/df9fb7c8121c498e89c5891ea713f5bd as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/df9fb7c8121c498e89c5891ea713f5bd 2023-04-28 14:57:09,022 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/df9fb7c8121c498e89c5891ea713f5bd, entries=150, sequenceid=303, filesize=11.9 K 2023-04-28 14:57:09,023 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/a7396c0b9cc946efb904be055e45f5bf as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/a7396c0b9cc946efb904be055e45f5bf 2023-04-28 14:57:09,029 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/a7396c0b9cc946efb904be055e45f5bf, entries=150, sequenceid=303, filesize=11.9 K 2023-04-28 14:57:09,031 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/9002a2ab62644ca08949a617d26ea42b as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/9002a2ab62644ca08949a617d26ea42b 2023-04-28 14:57:09,038 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/9002a2ab62644ca08949a617d26ea42b, entries=150, sequenceid=303, filesize=11.9 K 2023-04-28 14:57:09,040 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for cce278287965607462f53e33830eba7b in 952ms, sequenceid=303, compaction requested=true 2023-04-28 14:57:09,040 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:09,040 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:09,041 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:09,041 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:09,041 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,047 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:09,047 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:09,048 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:09,048 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:09,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:09,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:09,049 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:09,049 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:09,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:09,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:09,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:09,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:09,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45a4119b[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:09,051 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:09,051 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:09,052 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:09,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:09,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:09,052 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:09,052 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:09,052 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:09,053 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:09,053 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:09,054 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:09,054 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,054 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:09,054 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:09,054 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:09,054 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:09,054 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:09,055 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:09,055 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:09,056 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:09,056 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:09,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:09,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:09,056 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:09,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:09,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:09,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:09,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:09,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:09,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:09,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:09,058 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:09,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:09,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:09,059 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:09,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:09,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,061 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:09,061 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:09,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:09,062 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:09,062 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:09,062 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:09,062 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,062 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:09,062 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:09,062 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:09,062 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:09,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:09,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:09,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:09,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:09,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:09,062 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:09,062 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:09,163 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:09,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:09,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:09,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:09,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:09,166 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:09,166 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:09,166 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:09,166 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:09,167 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:09,167 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:09,167 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:09,167 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:09,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:09,168 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,168 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:09,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:09,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:09,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:09,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:09,168 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:09,169 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:09,169 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:09,169 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:09,169 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:09,169 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:09,169 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:09,169 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:09,169 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:09,169 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:09,169 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cce278287965607462f53e33830eba7b 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-28 14:57:09,169 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=A 2023-04-28 14:57:09,169 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:09,170 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=B 2023-04-28 14:57:09,170 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:09,170 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=C 2023-04-28 14:57:09,170 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:09,197 WARN [DataStreamer for file /user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/a794d60fddae4b528f6b1aae45a18639] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-28 14:57:09,197 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/a794d60fddae4b528f6b1aae45a18639 2023-04-28 14:57:09,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on cce278287965607462f53e33830eba7b 2023-04-28 14:57:09,258 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:09,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:09,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693889277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,283 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:09,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693889279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,285 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:09,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693889281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:09,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693889282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:09,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693889283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,385 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:09,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693889385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:09,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693889386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:09,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:09,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693889393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693889387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,395 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:09,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693889394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:09,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693889588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:09,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693889593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,601 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:09,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693889596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,602 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:09,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693889597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,602 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:09,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693889598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,621 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/e2abdb9125fc480e945bf7de5a09479b 2023-04-28 14:57:09,656 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/ea4ae34ca6cc4fcf861d674f345e2685 2023-04-28 14:57:09,667 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/a794d60fddae4b528f6b1aae45a18639 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/a794d60fddae4b528f6b1aae45a18639 2023-04-28 14:57:09,675 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/a794d60fddae4b528f6b1aae45a18639, entries=150, sequenceid=320, filesize=11.9 K 2023-04-28 14:57:09,682 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/e2abdb9125fc480e945bf7de5a09479b as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/e2abdb9125fc480e945bf7de5a09479b 2023-04-28 14:57:09,691 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/e2abdb9125fc480e945bf7de5a09479b, entries=150, sequenceid=320, filesize=11.9 K 2023-04-28 14:57:09,692 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/ea4ae34ca6cc4fcf861d674f345e2685 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/ea4ae34ca6cc4fcf861d674f345e2685 2023-04-28 14:57:09,705 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/ea4ae34ca6cc4fcf861d674f345e2685, entries=150, sequenceid=320, filesize=11.9 K 2023-04-28 14:57:09,706 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for cce278287965607462f53e33830eba7b in 537ms, sequenceid=320, compaction requested=true 2023-04-28 14:57:09,706 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:09,706 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:09,706 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:09,706 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:09,706 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,706 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:09,706 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:09,707 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:09,707 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:09,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:09,708 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:09,708 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:09,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:09,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:09,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:09,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:09,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:09,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6054cbdf[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:09,710 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:09,710 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:09,711 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:09,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:09,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:09,711 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:09,711 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:09,711 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,711 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:09,711 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:09,712 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:09,712 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,712 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:09,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:09,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:09,712 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:09,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:09,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:09,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:09,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:09,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:09,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:09,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:09,716 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:09,716 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:09,716 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:09,716 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:09,716 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:09,717 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:09,717 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:09,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:09,717 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:09,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:09,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:09,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:09,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:09,718 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:09,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:09,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:09,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:09,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:09,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:09,720 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:09,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:09,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:09,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,721 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:09,721 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:09,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:09,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:09,721 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:09,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:09,721 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:09,722 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:09,722 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:09,721 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:09,722 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:09,722 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,722 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:09,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:09,722 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:09,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:09,722 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:09,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:09,822 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:09,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:09,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:09,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:09,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:09,827 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:09,827 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:09,827 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:09,827 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:09,827 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:09,827 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:09,829 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:09,830 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:09,830 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,830 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:09,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:09,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:09,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:09,831 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:09,831 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:09,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:09,831 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:09,831 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:09,831 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:09,838 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:09,838 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:09,838 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:09,838 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:09,838 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cce278287965607462f53e33830eba7b 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-28 14:57:09,838 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=A 2023-04-28 14:57:09,839 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:09,839 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=B 2023-04-28 14:57:09,839 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:09,839 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=C 2023-04-28 14:57:09,839 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:09,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on cce278287965607462f53e33830eba7b 2023-04-28 14:57:09,896 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:09,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:09,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693889912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:09,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693889913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:09,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693889914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:09,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693889915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:09,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:09,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693889916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:10,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693890017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,021 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:10,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693890019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:10,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693890019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:10,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693890019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:10,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693890020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:10,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693890221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:10,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693890222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:10,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693890223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:10,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693890224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:10,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693890226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,251 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/ba04af465b314451aa1bf44fcce90f88 2023-04-28 14:57:10,266 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/5c428c352a35496ba5066c471891218e 2023-04-28 14:57:10,293 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/564671477c2842e0ac5ddfa9c9fdca0e 2023-04-28 14:57:10,300 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/ba04af465b314451aa1bf44fcce90f88 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/ba04af465b314451aa1bf44fcce90f88 2023-04-28 14:57:10,308 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/ba04af465b314451aa1bf44fcce90f88, entries=150, sequenceid=335, filesize=11.9 K 2023-04-28 14:57:10,309 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/5c428c352a35496ba5066c471891218e as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/5c428c352a35496ba5066c471891218e 2023-04-28 14:57:10,316 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/5c428c352a35496ba5066c471891218e, entries=150, sequenceid=335, filesize=11.9 K 2023-04-28 14:57:10,322 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/564671477c2842e0ac5ddfa9c9fdca0e as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/564671477c2842e0ac5ddfa9c9fdca0e 2023-04-28 14:57:10,333 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/564671477c2842e0ac5ddfa9c9fdca0e, entries=150, sequenceid=335, filesize=11.9 K 2023-04-28 14:57:10,333 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for cce278287965607462f53e33830eba7b in 495ms, sequenceid=335, compaction requested=true 2023-04-28 14:57:10,334 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:10,334 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:10,334 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:10,334 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:10,334 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,334 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:10,334 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:10,336 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:10,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:10,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:10,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:10,337 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:10,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:10,337 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:10,338 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:10,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:10,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:10,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6baa8efb[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:10,338 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:10,338 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:10,339 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:10,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:10,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:10,339 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,339 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:10,339 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:10,339 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:10,339 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:10,340 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:10,340 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,340 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:10,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:10,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:10,340 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:10,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:10,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:10,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:10,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:10,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:10,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:10,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:10,342 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:10,343 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:10,343 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:10,343 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:10,343 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:10,344 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:10,344 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:10,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:10,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:10,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:10,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:10,344 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:10,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:10,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:10,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:10,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:10,345 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:10,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:10,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:10,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:10,346 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:10,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:10,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,349 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,349 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:10,349 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:10,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:10,349 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:10,349 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:10,349 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:10,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:10,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:10,349 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:10,350 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:10,350 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:10,350 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:10,349 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:10,350 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:10,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:10,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:10,450 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:10,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:10,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:10,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:10,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:10,454 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:10,454 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:10,454 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:10,454 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:10,454 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:10,454 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:10,456 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:10,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:10,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:10,456 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:10,456 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,456 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:10,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:10,457 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:10,457 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:10,457 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:10,457 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:10,457 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:10,457 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:10,462 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:10,462 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:10,462 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:10,462 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:10,462 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cce278287965607462f53e33830eba7b 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-28 14:57:10,462 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=A 2023-04-28 14:57:10,462 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:10,462 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=B 2023-04-28 14:57:10,463 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:10,463 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=C 2023-04-28 14:57:10,463 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:10,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on cce278287965607462f53e33830eba7b 2023-04-28 14:57:10,528 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:10,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:10,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693890540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:10,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693890541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,548 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:10,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693890542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,548 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:10,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693890543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,548 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:10,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693890545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:10,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693890648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,653 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:10,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:10,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693890649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693890649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:10,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693890649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:10,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693890649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,861 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:10,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693890854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:10,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693890855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:10,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693890856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:10,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693890856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:10,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693890857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:10,889 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/0549671a248844f581fb4725194c32f1 2023-04-28 14:57:11,164 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:11,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693891164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:11,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:11,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693891164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:11,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:11,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693891164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:11,167 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:11,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693891165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:11,167 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:11,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693891166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:11,341 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/d19122c1c49c4547948113d298978a21 2023-04-28 14:57:11,357 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/09ce35b392494a8290ecad18442c98c0 2023-04-28 14:57:11,362 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/0549671a248844f581fb4725194c32f1 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/0549671a248844f581fb4725194c32f1 2023-04-28 14:57:11,367 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/0549671a248844f581fb4725194c32f1, entries=150, sequenceid=352, filesize=11.9 K 2023-04-28 14:57:11,368 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/d19122c1c49c4547948113d298978a21 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/d19122c1c49c4547948113d298978a21 2023-04-28 14:57:11,373 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/d19122c1c49c4547948113d298978a21, entries=150, sequenceid=352, filesize=11.9 K 2023-04-28 14:57:11,374 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/09ce35b392494a8290ecad18442c98c0 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/09ce35b392494a8290ecad18442c98c0 2023-04-28 14:57:11,379 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/09ce35b392494a8290ecad18442c98c0, entries=150, sequenceid=352, filesize=11.9 K 2023-04-28 14:57:11,380 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for cce278287965607462f53e33830eba7b in 918ms, sequenceid=352, compaction requested=true 2023-04-28 14:57:11,380 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:11,380 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:11,380 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:11,380 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:11,380 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:11,380 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:11,380 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:11,381 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:11,381 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:11,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:11,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:11,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:11,381 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:11,381 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:11,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:11,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:11,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:11,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:11,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:11,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:11,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42c79fc3[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:11,383 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:11,383 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:11,384 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:11,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:11,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:11,384 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:11,384 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:11,384 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:11,385 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:11,385 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:11,385 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:11,385 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:11,386 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:11,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:11,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:11,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:11,386 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:11,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:11,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:11,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:11,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:11,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:11,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:11,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:11,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:11,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:11,388 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:11,388 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:11,388 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:11,388 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:11,388 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:11,389 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:11,389 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:11,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:11,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:11,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:11,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:11,389 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:11,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:11,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:11,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:11,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:11,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:11,390 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:11,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:11,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:11,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:11,391 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:11,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:11,393 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:11,393 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:11,393 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:11,393 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:11,393 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:11,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:11,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:11,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:11,393 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:11,393 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:11,393 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:11,393 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:11,393 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:11,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:11,393 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:11,393 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:11,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:11,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:11,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:11,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:11,493 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:11,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:11,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:11,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:11,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:11,496 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:11,496 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:11,496 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:11,496 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:11,497 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:11,497 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:11,498 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:11,498 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:11,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:11,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:11,498 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:11,499 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:11,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:11,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:11,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:11,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:11,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:11,500 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:11,500 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:11,500 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:11,501 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:11,501 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:11,502 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:11,502 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:11,502 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cce278287965607462f53e33830eba7b 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-28 14:57:11,502 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=A 2023-04-28 14:57:11,502 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:11,503 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=B 2023-04-28 14:57:11,503 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:11,503 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=C 2023-04-28 14:57:11,503 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:11,526 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/3f6d55ba5f9e49b98c006f0828067516 2023-04-28 14:57:11,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on cce278287965607462f53e33830eba7b 2023-04-28 14:57:11,669 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:11,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:11,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693891690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:11,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:11,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693891690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:11,697 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:11,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693891692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:11,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:11,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:11,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693891695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:11,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693891696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:11,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:11,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693891798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:11,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:11,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693891798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:11,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:11,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693891798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:11,807 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:11,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693891805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:11,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:11,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693891807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:11,953 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/f43290f08a5c426db452ea946cad317a 2023-04-28 14:57:12,005 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:12,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693892004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:12,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693892006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:12,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693892006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,011 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:12,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693892008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:12,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693892012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:12,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693892307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:12,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693892313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:12,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693892313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:12,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693892318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:12,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693892319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,371 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/a415c8043f54483bba57b839146993bd 2023-04-28 14:57:12,380 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/3f6d55ba5f9e49b98c006f0828067516 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/3f6d55ba5f9e49b98c006f0828067516 2023-04-28 14:57:12,386 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/3f6d55ba5f9e49b98c006f0828067516, entries=150, sequenceid=367, filesize=11.9 K 2023-04-28 14:57:12,387 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/f43290f08a5c426db452ea946cad317a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/f43290f08a5c426db452ea946cad317a 2023-04-28 14:57:12,394 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/f43290f08a5c426db452ea946cad317a, entries=150, sequenceid=367, filesize=11.9 K 2023-04-28 14:57:12,395 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/a415c8043f54483bba57b839146993bd as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/a415c8043f54483bba57b839146993bd 2023-04-28 14:57:12,414 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/a415c8043f54483bba57b839146993bd, entries=150, sequenceid=367, filesize=11.9 K 2023-04-28 14:57:12,415 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for cce278287965607462f53e33830eba7b in 913ms, sequenceid=367, compaction requested=true 2023-04-28 14:57:12,415 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:12,415 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:12,415 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:12,415 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:12,415 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,416 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:12,416 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:12,417 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:12,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:12,417 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:12,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:12,417 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:12,417 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:12,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:12,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:12,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:12,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:12,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@278a76c9[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:12,422 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:12,422 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:12,423 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:12,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:12,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:12,424 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:12,424 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:12,424 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:12,425 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:12,425 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:12,425 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:12,426 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,426 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:12,427 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:12,427 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:12,427 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:12,427 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:12,427 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:12,427 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:12,429 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:12,429 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:12,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:12,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:12,429 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:12,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:12,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:12,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:12,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:12,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:12,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:12,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:12,430 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,431 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:12,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:12,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,433 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:12,434 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:12,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:12,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:12,434 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,434 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:12,434 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:12,434 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:12,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:12,434 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:12,434 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:12,434 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:12,434 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:12,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:12,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:12,435 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:12,435 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:12,535 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:12,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:12,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:12,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:12,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:12,538 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:12,538 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:12,538 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:12,538 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:12,539 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:12,539 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:12,539 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:12,539 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:12,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:12,540 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,540 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:12,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:12,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:12,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:12,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:12,541 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:12,541 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:12,541 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:12,541 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:12,541 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:12,541 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:12,541 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:12,541 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:12,541 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:12,542 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:12,542 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cce278287965607462f53e33830eba7b 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-28 14:57:12,542 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=A 2023-04-28 14:57:12,542 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:12,542 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=B 2023-04-28 14:57:12,542 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:12,542 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=C 2023-04-28 14:57:12,542 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:12,573 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/12855529b6cb4d938d447a48680a4ee5 2023-04-28 14:57:12,593 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/6c02d9c58fdd4ff2a8c11d0ec8829970 2023-04-28 14:57:12,644 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/af3ea4cb90a2466aa4d09f1ed5aba1b7 2023-04-28 14:57:12,658 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/12855529b6cb4d938d447a48680a4ee5 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/12855529b6cb4d938d447a48680a4ee5 2023-04-28 14:57:12,666 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/12855529b6cb4d938d447a48680a4ee5, entries=150, sequenceid=384, filesize=11.9 K 2023-04-28 14:57:12,668 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/6c02d9c58fdd4ff2a8c11d0ec8829970 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/6c02d9c58fdd4ff2a8c11d0ec8829970 2023-04-28 14:57:12,679 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/6c02d9c58fdd4ff2a8c11d0ec8829970, entries=150, sequenceid=384, filesize=11.9 K 2023-04-28 14:57:12,682 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/af3ea4cb90a2466aa4d09f1ed5aba1b7 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/af3ea4cb90a2466aa4d09f1ed5aba1b7 2023-04-28 14:57:12,688 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/af3ea4cb90a2466aa4d09f1ed5aba1b7, entries=150, sequenceid=384, filesize=11.9 K 2023-04-28 14:57:12,690 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for cce278287965607462f53e33830eba7b in 148ms, sequenceid=384, compaction requested=true 2023-04-28 14:57:12,690 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:12,690 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:12,690 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:12,690 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:12,690 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,690 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:12,690 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:12,692 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,693 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:12,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:12,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:12,694 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:12,694 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:12,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:12,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61464bd4[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:12,695 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:12,695 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:12,696 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:12,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:12,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:12,696 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:12,696 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:12,696 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:12,697 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,697 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:12,697 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,697 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:12,697 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:12,697 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:12,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:12,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:12,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:12,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:12,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:12,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:12,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:12,707 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:12,707 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:12,707 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:12,707 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:12,707 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:12,709 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:12,709 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:12,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:12,709 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:12,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:12,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:12,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:12,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:12,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:12,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:12,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:12,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:12,710 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:12,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:12,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:12,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,712 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:12,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:12,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,717 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:12,717 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,717 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:12,717 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:12,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:12,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:12,717 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:12,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:12,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:12,717 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:12,717 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:12,717 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:12,717 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,717 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:12,718 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:12,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:12,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:12,718 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:12,817 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:12,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:12,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:12,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:12,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:12,824 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:12,824 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:12,824 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:12,824 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:12,825 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:12,825 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:12,827 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:12,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:12,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:12,827 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:12,829 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,829 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:12,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:12,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:12,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:12,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:12,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on cce278287965607462f53e33830eba7b 2023-04-28 14:57:12,836 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cce278287965607462f53e33830eba7b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 14:57:12,836 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=A 2023-04-28 14:57:12,836 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:12,837 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=B 2023-04-28 14:57:12,837 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:12,837 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=C 2023-04-28 14:57:12,837 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:12,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:12,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:12,838 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:12,838 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:12,838 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:12,850 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:12,850 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:12,855 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:12,855 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:12,855 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:12,856 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:12,856 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:12,856 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:12,856 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,856 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:12,856 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:12,859 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,859 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:12,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:12,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:12,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:12,861 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:12,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:12,861 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:12,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:12,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:12,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:12,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ea4dc51[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:12,866 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:12,866 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:12,867 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:12,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:12,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:12,867 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:12,867 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:12,867 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:12,868 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:12,868 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:12,868 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:12,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682693832851, max timestamp=1682693832871Num uniques 859; 2023-04-28 14:57:12,874 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 14:57:12,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682693832851, max timestamp=1682693832871Num uniques 859; 2023-04-28 14:57:12,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 14:57:12,874 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:12,874 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:12,874 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:12,874 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:12,874 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:12,874 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:12,874 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:12,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682693832851, max timestamp=1682693832871Num uniques 859; 2023-04-28 14:57:12,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 14:57:12,876 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:12,876 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:12,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:12,876 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/e485cbd54be0476999aba2cd9e16881b 2023-04-28 14:57:12,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:12,877 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,877 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:12,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:12,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:12,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:12,879 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:12,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:12,882 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:12,882 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:12,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:12,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:12,883 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,883 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:12,883 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:12,883 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,883 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:12,883 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:12,883 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:12,883 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:12,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:12,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:12,884 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:12,884 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:12,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:12,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:12,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:12,899 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/f5c42110a008444da4d6111237363e3d 2023-04-28 14:57:12,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:12,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693892897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:12,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693892899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:12,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693892899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,917 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:12,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693892908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,917 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:12,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693892909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,984 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:12,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:12,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:12,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:12,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:12,987 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:12,987 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:12,987 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:12,987 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:12,988 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:12,988 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:12,989 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:12,989 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:12,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:12,989 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,989 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:12,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:12,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:12,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:12,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:12,990 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:12,994 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:12,994 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:12,994 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:12,994 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:12,994 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:12,994 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:12,994 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:12,994 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:12,994 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:12,995 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:12,995 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:12,995 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:12,995 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:12,995 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,995 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:12,995 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:12,996 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,996 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:12,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:12,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:12,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:12,999 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:12,999 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:13,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:13,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:13,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:13,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:13,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:13,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20e379c1[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:13,004 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:13,004 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:13,005 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:13,005 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,005 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:13,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:13,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:13,006 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:13,006 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:13,006 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:13,007 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:13,007 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:13,007 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,007 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:13,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:13,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:13,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:13,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:13,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:13,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:13,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:13,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:13,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:13,010 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:13,010 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:13,010 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:13,010 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:13,010 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:13,012 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:13,012 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:13,012 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:13,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:13,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:13,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:13,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:13,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:13,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:13,013 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:13,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:13,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:13,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:13,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:13,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:13,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,015 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:13,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:13,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,017 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:13,017 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:13,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:13,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:13,017 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,017 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:13,017 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:13,017 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,018 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:13,018 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:13,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:13,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:13,018 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:13,018 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:13,018 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:13,018 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:13,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:13,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:13,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:13,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:13,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693893011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:13,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693893014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:13,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693893015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:13,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693893018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:13,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693893019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,118 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:13,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:13,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:13,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:13,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:13,133 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:13,133 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:13,133 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:13,133 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:13,133 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:13,133 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:13,134 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:13,134 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:13,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:13,134 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:13,134 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:13,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:13,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:13,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:13,135 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:13,138 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:13,138 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:13,138 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:13,139 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:13,139 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:13,142 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:13,142 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:13,142 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:13,142 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:13,142 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:13,142 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:13,142 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:13,142 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:13,142 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,142 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:13,142 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:13,143 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,143 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:13,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:13,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:13,144 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:13,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:13,144 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:13,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:13,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:13,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:13,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:13,146 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:13,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54ab6f2a[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:13,146 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:13,146 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:13,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:13,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:13,147 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:13,147 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:13,147 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:13,150 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,150 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:13,151 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,151 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:13,151 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:13,151 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:13,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:13,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:13,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:13,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:13,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:13,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:13,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:13,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:13,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:13,154 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:13,154 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:13,154 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:13,154 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:13,155 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:13,155 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:13,156 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:13,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:13,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:13,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:13,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:13,156 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:13,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:13,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:13,156 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:13,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:13,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:13,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:13,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:13,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:13,157 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:13,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:13,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,159 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,159 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:13,159 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:13,159 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,159 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:13,159 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:13,159 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:13,159 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:13,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:13,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:13,159 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:13,159 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:13,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:13,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:13,160 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:13,160 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:13,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:13,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:13,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:13,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693893223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:13,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693893224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,231 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:13,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693893225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,231 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:13,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693893226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,231 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:13,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693893226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,260 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:13,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:13,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:13,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:13,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:13,266 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:13,266 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:13,266 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:13,266 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:13,267 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:13,267 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:13,268 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,268 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:13,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:13,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:13,268 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,268 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:13,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:13,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:13,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:13,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:13,269 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:13,269 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:13,269 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:13,269 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:13,269 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:13,270 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:13,270 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:13,270 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:13,270 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:13,270 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:13,270 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:13,270 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:13,270 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:13,270 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:13,270 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,270 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:13,270 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:13,275 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:13,275 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:13,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:13,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:13,276 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:13,276 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:13,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:13,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:13,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:13,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:13,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7608680[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:13,279 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:13,279 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:13,280 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:13,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:13,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:13,281 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:13,281 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:13,281 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:13,281 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,281 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:13,282 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:13,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:13,283 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:13,283 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:13,283 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:13,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:13,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:13,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:13,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:13,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:13,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:13,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:13,288 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:13,288 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:13,288 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:13,288 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:13,288 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:13,289 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:13,289 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:13,289 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:13,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:13,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:13,290 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:13,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:13,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:13,291 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:13,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:13,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,294 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,294 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:13,294 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:13,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:13,294 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:13,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:13,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:13,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:13,296 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:13,296 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,296 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:13,296 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:13,296 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:13,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:13,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:13,296 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:13,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:13,296 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:13,296 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:13,337 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/5120d6cfc9024c939c163044613b68ec 2023-04-28 14:57:13,344 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/e485cbd54be0476999aba2cd9e16881b as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/e485cbd54be0476999aba2cd9e16881b 2023-04-28 14:57:13,354 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/e485cbd54be0476999aba2cd9e16881b, entries=250, sequenceid=394, filesize=16.7 K 2023-04-28 14:57:13,368 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/f5c42110a008444da4d6111237363e3d as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/f5c42110a008444da4d6111237363e3d 2023-04-28 14:57:13,376 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/f5c42110a008444da4d6111237363e3d, entries=150, sequenceid=394, filesize=11.9 K 2023-04-28 14:57:13,377 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/5120d6cfc9024c939c163044613b68ec as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/5120d6cfc9024c939c163044613b68ec 2023-04-28 14:57:13,398 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:13,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:13,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:13,416 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/5120d6cfc9024c939c163044613b68ec, entries=150, sequenceid=394, filesize=11.9 K 2023-04-28 14:57:13,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:13,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:13,416 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:13,416 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:13,416 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:13,416 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:13,417 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for cce278287965607462f53e33830eba7b in 581ms, sequenceid=394, compaction requested=true 2023-04-28 14:57:13,417 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:13,417 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:13,417 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:57:13,417 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 10 store files, 0 compacting, 10 eligible, 16 blocking 2023-04-28 14:57:13,417 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 10 store files, 0 compacting, 10 eligible, 16 blocking 2023-04-28 14:57:13,417 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:57:13,417 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:13,417 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:13,423 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 127018 starting at candidate #0 after considering 36 permutations with 36 in ratio 2023-04-28 14:57:13,423 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): cce278287965607462f53e33830eba7b/A is initiating minor compaction (all files) 2023-04-28 14:57:13,423 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cce278287965607462f53e33830eba7b/A in TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:13,423 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/27f76f301fbc4f52adcf1ac9dc93cdb2, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/83455c337ff4444bb4b2764de10b95fb, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/92e9978209b4410bb869cde95a40ce9e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/df9fb7c8121c498e89c5891ea713f5bd, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/a794d60fddae4b528f6b1aae45a18639, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/ba04af465b314451aa1bf44fcce90f88, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/0549671a248844f581fb4725194c32f1, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/3f6d55ba5f9e49b98c006f0828067516, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/12855529b6cb4d938d447a48680a4ee5, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/e485cbd54be0476999aba2cd9e16881b] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp, totalSize=124.0 K 2023-04-28 14:57:13,425 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 122138 starting at candidate #0 after considering 36 permutations with 36 in ratio 2023-04-28 14:57:13,425 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): cce278287965607462f53e33830eba7b/B is initiating minor compaction (all files) 2023-04-28 14:57:13,425 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cce278287965607462f53e33830eba7b/B in TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:13,425 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/cb01a112c018439e90757ec6069a94ad, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/f498f8c613cb4ce1810a37526b69b17a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/b708905bbf4849a6a39bfae2827e9a8b, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/a7396c0b9cc946efb904be055e45f5bf, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/e2abdb9125fc480e945bf7de5a09479b, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/5c428c352a35496ba5066c471891218e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/d19122c1c49c4547948113d298978a21, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/f43290f08a5c426db452ea946cad317a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/6c02d9c58fdd4ff2a8c11d0ec8829970, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/f5c42110a008444da4d6111237363e3d] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp, totalSize=119.3 K 2023-04-28 14:57:13,425 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 27f76f301fbc4f52adcf1ac9dc93cdb2, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1682693824557 2023-04-28 14:57:13,426 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting cb01a112c018439e90757ec6069a94ad, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1682693824557 2023-04-28 14:57:13,426 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 83455c337ff4444bb4b2764de10b95fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1682693825185 2023-04-28 14:57:13,426 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting f498f8c613cb4ce1810a37526b69b17a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1682693825185 2023-04-28 14:57:13,426 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 92e9978209b4410bb869cde95a40ce9e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1682693826338 2023-04-28 14:57:13,427 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:13,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:13,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:13,427 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:13,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:13,427 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,427 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:13,428 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting b708905bbf4849a6a39bfae2827e9a8b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1682693826338 2023-04-28 14:57:13,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:13,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:13,428 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting df9fb7c8121c498e89c5891ea713f5bd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1682693826971 2023-04-28 14:57:13,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:13,428 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting a794d60fddae4b528f6b1aae45a18639, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1682693828129 2023-04-28 14:57:13,429 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting ba04af465b314451aa1bf44fcce90f88, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1682693829277 2023-04-28 14:57:13,429 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 0549671a248844f581fb4725194c32f1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1682693829905 2023-04-28 14:57:13,430 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting a7396c0b9cc946efb904be055e45f5bf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1682693826971 2023-04-28 14:57:13,430 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 3f6d55ba5f9e49b98c006f0828067516, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1682693830540 2023-04-28 14:57:13,430 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting e2abdb9125fc480e945bf7de5a09479b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1682693828129 2023-04-28 14:57:13,430 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 12855529b6cb4d938d447a48680a4ee5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1682693831677 2023-04-28 14:57:13,430 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 5c428c352a35496ba5066c471891218e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1682693829277 2023-04-28 14:57:13,431 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting e485cbd54be0476999aba2cd9e16881b, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1682693832822 2023-04-28 14:57:13,431 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting d19122c1c49c4547948113d298978a21, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1682693829905 2023-04-28 14:57:13,431 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting f43290f08a5c426db452ea946cad317a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1682693830540 2023-04-28 14:57:13,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:13,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:13,431 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:13,431 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:13,431 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:13,432 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 6c02d9c58fdd4ff2a8c11d0ec8829970, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1682693831677 2023-04-28 14:57:13,432 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:13,432 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:13,432 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:13,432 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting f5c42110a008444da4d6111237363e3d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1682693832824 2023-04-28 14:57:13,432 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:13,432 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cce278287965607462f53e33830eba7b 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-28 14:57:13,432 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=A 2023-04-28 14:57:13,433 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:13,433 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=B 2023-04-28 14:57:13,433 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:13,433 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=C 2023-04-28 14:57:13,433 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:13,490 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): cce278287965607462f53e33830eba7b#A#compaction#73 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:13,500 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): cce278287965607462f53e33830eba7b#B#compaction#74 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:13,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on cce278287965607462f53e33830eba7b 2023-04-28 14:57:13,532 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:13,543 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/9099770035b74103a5919a15d09f65a6 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/9099770035b74103a5919a15d09f65a6 2023-04-28 14:57:13,601 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 10 (all) file(s) in cce278287965607462f53e33830eba7b/B of cce278287965607462f53e33830eba7b into 9099770035b74103a5919a15d09f65a6(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:13,602 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:13,602 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b., storeName=cce278287965607462f53e33830eba7b/B, priority=6, startTime=1682693833417; duration=0sec 2023-04-28 14:57:13,602 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:57:13,602 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 10 store files, 0 compacting, 10 eligible, 16 blocking 2023-04-28 14:57:13,606 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 122138 starting at candidate #0 after considering 36 permutations with 36 in ratio 2023-04-28 14:57:13,606 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): cce278287965607462f53e33830eba7b/C is initiating minor compaction (all files) 2023-04-28 14:57:13,606 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cce278287965607462f53e33830eba7b/C in TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:13,606 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/18c8c5124a7b436e8214367177e7761e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/ba43580190f5438b98af185d3ef061c5, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/737e68e8f0cf40cf8ba35195a357f97c, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/9002a2ab62644ca08949a617d26ea42b, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/ea4ae34ca6cc4fcf861d674f345e2685, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/564671477c2842e0ac5ddfa9c9fdca0e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/09ce35b392494a8290ecad18442c98c0, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/a415c8043f54483bba57b839146993bd, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/af3ea4cb90a2466aa4d09f1ed5aba1b7, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/5120d6cfc9024c939c163044613b68ec] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp, totalSize=119.3 K 2023-04-28 14:57:13,607 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 18c8c5124a7b436e8214367177e7761e, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1682693824557 2023-04-28 14:57:13,607 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting ba43580190f5438b98af185d3ef061c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1682693825185 2023-04-28 14:57:13,611 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 737e68e8f0cf40cf8ba35195a357f97c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1682693826338 2023-04-28 14:57:13,612 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 9002a2ab62644ca08949a617d26ea42b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1682693826971 2023-04-28 14:57:13,612 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting ea4ae34ca6cc4fcf861d674f345e2685, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1682693828129 2023-04-28 14:57:13,613 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 564671477c2842e0ac5ddfa9c9fdca0e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1682693829277 2023-04-28 14:57:13,613 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 09ce35b392494a8290ecad18442c98c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1682693829905 2023-04-28 14:57:13,613 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting a415c8043f54483bba57b839146993bd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1682693830540 2023-04-28 14:57:13,614 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting af3ea4cb90a2466aa4d09f1ed5aba1b7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1682693831677 2023-04-28 14:57:13,614 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 5120d6cfc9024c939c163044613b68ec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1682693832824 2023-04-28 14:57:13,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:13,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693893603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,617 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:13,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693893603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,617 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:13,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693893604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:13,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693893605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:13,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693893605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,652 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): cce278287965607462f53e33830eba7b#C#compaction#75 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:13,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:13,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693893718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:13,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693893718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:13,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693893718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:13,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693893719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:13,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693893719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,913 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/022f0dc399f24d7c948e0a6c81ddfab9 2023-04-28 14:57:13,922 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:13,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693893922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:13,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693893922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,925 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:13,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693893922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,925 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:13,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693893924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,926 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:13,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693893925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:13,933 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/8282f02cb18d4f668fc68753ffd98afe 2023-04-28 14:57:13,939 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/b6a4a733e0954b40a7af8ef6c310f9a3 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/b6a4a733e0954b40a7af8ef6c310f9a3 2023-04-28 14:57:13,948 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 10 (all) file(s) in cce278287965607462f53e33830eba7b/A of cce278287965607462f53e33830eba7b into b6a4a733e0954b40a7af8ef6c310f9a3(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:13,948 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:13,948 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b., storeName=cce278287965607462f53e33830eba7b/A, priority=6, startTime=1682693833417; duration=0sec 2023-04-28 14:57:13,948 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:14,083 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/f5583a948f85434bbfee3da773638a00 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/f5583a948f85434bbfee3da773638a00 2023-04-28 14:57:14,091 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 10 (all) file(s) in cce278287965607462f53e33830eba7b/C of cce278287965607462f53e33830eba7b into f5583a948f85434bbfee3da773638a00(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:14,092 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:14,092 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b., storeName=cce278287965607462f53e33830eba7b/C, priority=6, startTime=1682693833417; duration=0sec 2023-04-28 14:57:14,092 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:14,228 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:14,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693894226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:14,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:14,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:14,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693894226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:14,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693894227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:14,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:14,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693894229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:14,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:14,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693894230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:14,368 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/10a2d798647142d1958a56ca1bca2b56 2023-04-28 14:57:14,382 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/022f0dc399f24d7c948e0a6c81ddfab9 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/022f0dc399f24d7c948e0a6c81ddfab9 2023-04-28 14:57:14,390 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/022f0dc399f24d7c948e0a6c81ddfab9, entries=150, sequenceid=429, filesize=11.9 K 2023-04-28 14:57:14,391 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/8282f02cb18d4f668fc68753ffd98afe as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/8282f02cb18d4f668fc68753ffd98afe 2023-04-28 14:57:14,399 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/8282f02cb18d4f668fc68753ffd98afe, entries=150, sequenceid=429, filesize=11.9 K 2023-04-28 14:57:14,400 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/10a2d798647142d1958a56ca1bca2b56 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/10a2d798647142d1958a56ca1bca2b56 2023-04-28 14:57:14,406 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/10a2d798647142d1958a56ca1bca2b56, entries=150, sequenceid=429, filesize=11.9 K 2023-04-28 14:57:14,408 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for cce278287965607462f53e33830eba7b in 976ms, sequenceid=429, compaction requested=false 2023-04-28 14:57:14,408 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:14,408 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:14,409 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:14,409 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:14,409 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:14,409 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:14,409 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:14,410 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:14,410 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:14,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:14,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:14,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:14,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:14,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:14,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:14,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:14,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:14,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:14,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fde9469[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:14,412 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:14,412 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:14,412 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:14,412 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:14,412 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:14,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:14,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:14,412 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:14,412 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:14,413 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:14,413 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:14,413 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:14,413 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:14,414 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:14,414 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:14,414 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:14,414 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:14,414 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:14,414 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:14,414 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:14,414 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:14,416 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:14,416 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:14,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:14,416 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:14,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:14,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:14,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:14,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:14,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:14,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:14,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:14,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:14,417 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:14,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:14,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:14,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:14,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:14,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:14,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:14,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:14,421 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:14,425 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:14,425 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:14,425 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:14,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:14,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:14,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:14,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:14,425 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:14,426 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:14,426 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:14,426 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:14,426 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:14,426 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:14,426 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:14,426 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:14,426 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:14,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:14,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:14,526 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:14,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:14,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:14,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:14,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:14,530 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:14,530 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:14,530 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:14,530 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:14,530 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:14,530 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:14,531 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:14,531 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:14,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:14,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:14,531 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:14,531 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:14,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:14,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:14,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:14,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:14,533 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:14,536 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:14,536 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:14,536 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:14,536 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:14,536 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:14,542 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:14,542 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:14,542 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:14,542 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:14,542 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cce278287965607462f53e33830eba7b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 14:57:14,542 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=A 2023-04-28 14:57:14,543 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:14,543 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=B 2023-04-28 14:57:14,543 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:14,543 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=C 2023-04-28 14:57:14,543 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:14,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on cce278287965607462f53e33830eba7b 2023-04-28 14:57:14,734 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:14,767 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:14,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693894765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:14,768 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:14,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693894766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:14,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693834763, max timestamp=1682693834766Num uniques 859; 2023-04-28 14:57:14,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:57:14,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693834763, max timestamp=1682693834766Num uniques 859; 2023-04-28 14:57:14,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693834763, max timestamp=1682693834766Num uniques 859; 2023-04-28 14:57:14,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:57:14,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:57:14,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682693834777, max timestamp=1682693834788Num uniques 1018; 2023-04-28 14:57:14,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682693834777, max timestamp=1682693834788Num uniques 1018; 2023-04-28 14:57:14,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-28 14:57:14,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-28 14:57:14,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682693834777, max timestamp=1682693834788Num uniques 1018; 2023-04-28 14:57:14,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-28 14:57:14,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:14,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693894809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:14,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:14,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693894809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:14,813 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:14,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693894810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:14,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:14,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693894869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:14,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:14,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693894869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:14,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:14,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693894913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:14,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:14,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693894913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:14,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:14,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693894914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:14,953 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/008e3f186fa248b1b15b69fd6fc3448f 2023-04-28 14:57:15,074 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:15,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693895071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:15,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693895073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,121 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:15,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693895117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,121 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:15,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693895118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,121 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:15,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693895118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,376 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/845038d42985407f8e8d99a90a93fadb 2023-04-28 14:57:15,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:15,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693895375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:15,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693895376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,425 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:15,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693895424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,426 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:15,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693895424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,427 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:15,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693895426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,787 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/bfce4cd51e414cc9a28a84eb2275dfc1 2023-04-28 14:57:15,793 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/008e3f186fa248b1b15b69fd6fc3448f as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/008e3f186fa248b1b15b69fd6fc3448f 2023-04-28 14:57:15,797 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/008e3f186fa248b1b15b69fd6fc3448f, entries=150, sequenceid=442, filesize=11.9 K 2023-04-28 14:57:15,798 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/845038d42985407f8e8d99a90a93fadb as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/845038d42985407f8e8d99a90a93fadb 2023-04-28 14:57:15,803 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/845038d42985407f8e8d99a90a93fadb, entries=150, sequenceid=442, filesize=11.9 K 2023-04-28 14:57:15,804 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/bfce4cd51e414cc9a28a84eb2275dfc1 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/bfce4cd51e414cc9a28a84eb2275dfc1 2023-04-28 14:57:15,808 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/bfce4cd51e414cc9a28a84eb2275dfc1, entries=150, sequenceid=442, filesize=11.9 K 2023-04-28 14:57:15,809 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for cce278287965607462f53e33830eba7b in 1267ms, sequenceid=442, compaction requested=true 2023-04-28 14:57:15,809 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:15,809 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:15,809 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:15,809 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:15,809 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,809 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:15,810 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:15,810 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,810 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:15,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:15,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:15,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:15,811 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:15,811 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:15,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:15,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:15,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:15,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:15,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4448d722[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:15,812 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:15,812 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:15,812 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:15,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:15,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:15,813 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:15,813 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:15,813 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,813 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:15,813 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:15,813 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:15,814 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,814 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:15,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:15,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:15,814 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:15,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:15,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:15,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:15,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:15,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:15,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:15,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:15,816 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:15,816 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:15,816 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:15,816 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:15,816 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:15,818 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:15,818 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:15,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:15,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:15,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:15,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:15,818 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:15,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:15,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:15,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:15,818 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:15,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:15,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:15,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:15,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:15,819 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:15,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:15,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,821 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,821 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:15,821 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:15,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:15,821 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:15,821 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:15,821 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:15,821 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:15,821 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:15,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:15,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:15,821 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:15,821 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:15,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:15,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:15,821 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:15,822 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:15,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on cce278287965607462f53e33830eba7b 2023-04-28 14:57:15,881 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cce278287965607462f53e33830eba7b 3/3 column families, dataSize=140.89 KB heapSize=394.69 KB 2023-04-28 14:57:15,881 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=A 2023-04-28 14:57:15,882 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:15,882 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=B 2023-04-28 14:57:15,882 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:15,882 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=C 2023-04-28 14:57:15,882 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:15,895 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:15,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693895890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:15,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693895895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,922 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:15,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:15,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:15,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:15,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:15,925 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:15,925 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:15,925 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:15,925 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:15,925 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:15,925 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:15,926 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,926 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,926 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:15,926 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:15,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:15,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:15,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:15,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:15,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:15,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:15,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:15,928 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:15,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693895928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,929 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:15,929 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:15,929 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:15,929 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:15,929 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:15,929 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:15,929 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:15,929 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:15,929 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:15,929 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:15,929 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:15,929 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:15,929 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:15,929 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,929 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:15,929 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:15,930 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:15,930 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:15,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:15,931 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:15,931 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:15,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:15,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:15,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:15,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:15,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:15,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b1532be[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:15,932 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:15,932 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:15,933 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:15,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:15,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:15,933 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:15,933 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:15,933 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:15,933 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,933 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:15,934 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,934 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:15,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:15,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:15,934 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:15,934 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:15,934 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:15,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693895929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:15,934 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:15,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693895930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:15,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:15,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:15,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:15,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:15,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:15,936 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:15,936 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:15,936 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:15,936 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:15,936 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:15,937 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:15,937 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:15,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:15,937 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:15,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:15,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:15,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:15,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:15,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:15,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:15,937 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:15,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:15,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:15,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:15,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:15,939 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:15,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:15,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,940 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,940 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:15,940 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:15,940 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:15,940 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:15,940 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:15,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:15,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:15,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:15,940 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:15,940 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:15,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:15,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:15,941 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:15,941 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:15,940 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:15,941 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:15,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:15,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:15,996 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:15,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693895996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,001 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:16,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693895998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,041 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:16,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:16,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:16,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:16,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:16,045 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:16,045 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:16,045 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:16,045 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:16,045 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,045 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,046 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,047 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:16,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:16,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:16,047 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,047 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:16,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:16,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,048 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:16,054 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:16,054 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:16,054 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:16,054 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:16,054 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:16,055 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:16,055 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:16,055 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:16,062 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:16,062 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:16,062 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:16,062 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:16,062 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:16,062 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,062 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:16,062 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:16,063 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,063 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:16,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:16,064 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,064 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:16,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:16,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:16,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:16,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:16,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4842f9da[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:16,066 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:16,066 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,069 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,070 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:16,070 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:16,070 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:16,071 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,071 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:16,071 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,071 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:16,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:16,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:16,071 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:16,072 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:16,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:16,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:16,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:16,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:16,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:16,074 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:16,074 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:16,074 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:16,074 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:16,075 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:16,076 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,076 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:16,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,076 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:16,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:16,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:16,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:16,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:16,077 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:16,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,081 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:16,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,086 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:16,086 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:16,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:16,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:16,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:16,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:16,087 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,087 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,087 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,087 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,087 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,087 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,087 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,087 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:16,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:16,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:16,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:16,088 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:16,088 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:16,188 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:16,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:16,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:16,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:16,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:16,192 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:16,192 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:16,192 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:16,192 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:16,193 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,193 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,193 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:16,193 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:16,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:16,194 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,194 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:16,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:16,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,194 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:16,195 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:16,195 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:16,195 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:16,195 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:16,195 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:16,195 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:16,195 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:16,195 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:16,195 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:16,195 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:16,195 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:16,195 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:16,195 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:16,195 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,195 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:16,195 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:16,196 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,196 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,196 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:16,197 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:16,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:16,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:16,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:16,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:16,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:16,198 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:16,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43344167[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:16,198 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,199 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,199 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:16,199 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:16,199 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,199 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:16,199 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:16,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:16,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693896199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,200 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:16,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:16,200 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:16,200 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:16,200 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:16,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:16,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:16,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:16,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:16,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:16,202 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:16,203 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:16,203 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:16,203 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:16,203 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:16,203 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:16,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693896203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,204 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,204 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,204 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:16,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:16,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:16,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:16,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:16,204 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:16,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:16,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,205 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:16,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,207 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:16,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:16,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,208 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:16,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:16,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:16,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:16,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:16,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:16,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:16,208 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:16,208 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:16,293 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/902c3a05af434d02b6a98cdfaf34c46d 2023-04-28 14:57:16,308 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:16,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:16,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:16,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:16,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:16,311 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:16,311 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:16,311 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:16,311 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:16,311 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/7dfc3ab7c3e540788967804ddeb78c11 2023-04-28 14:57:16,312 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,312 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,313 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:16,313 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:16,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:16,313 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,313 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:16,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:16,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:16,321 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:16,321 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:16,321 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:16,321 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:16,321 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:16,338 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:16,338 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:16,339 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:16,339 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:16,339 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:16,339 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:16,339 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:16,339 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:16,339 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,339 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:16,339 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:16,341 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:16,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:16,341 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:16,342 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,342 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:16,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:16,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:16,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:16,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41c76fe4[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:16,343 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:16,343 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,344 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,344 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:16,344 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:16,344 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:16,344 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,344 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:16,345 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:16,345 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,345 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:16,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:16,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:16,345 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:16,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:16,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:16,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:16,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:16,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:16,347 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:16,347 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:16,347 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:16,348 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:16,348 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:16,349 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:16,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:16,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:16,349 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,349 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,349 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:16,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:16,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:16,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:16,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,350 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:16,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,352 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,352 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,352 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,352 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,352 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:16,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:16,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:16,352 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:16,352 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:16,352 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,353 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:16,353 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:16,352 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:16,353 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:16,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:16,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:16,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:16,453 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:16,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:16,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:16,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:16,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:16,456 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:16,456 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:16,456 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:16,456 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:16,456 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,457 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,457 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,457 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:16,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:16,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:16,457 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,457 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:16,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:16,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:16,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:16,458 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:16,458 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:16,458 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:16,459 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:16,459 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:16,459 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:16,459 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:16,459 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:16,459 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:16,459 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:16,459 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:16,459 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:16,459 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,459 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:16,459 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:16,460 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,460 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:16,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:16,460 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,460 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:16,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:16,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:16,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:16,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:16,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@232535a0[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:16,462 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:16,462 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,463 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,463 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:16,463 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:16,463 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:16,463 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,463 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:16,464 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,464 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:16,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:16,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:16,464 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:16,464 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:16,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:16,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:16,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:16,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:16,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:16,466 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:16,466 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:16,466 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:16,466 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:16,466 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:16,467 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,467 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:16,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,467 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:16,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:16,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:16,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:16,467 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:16,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:16,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,469 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:16,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,470 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,470 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,470 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,470 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,470 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,470 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,470 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,470 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:16,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:16,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:16,470 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:16,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:16,471 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:16,471 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:16,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:16,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:16,470 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:16,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:16,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:16,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:16,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693896503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:16,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693896506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,571 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:16,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:16,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:16,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:16,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:16,574 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:16,574 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:16,574 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:16,574 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:16,575 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,575 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,577 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,577 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:16,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:16,577 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,577 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:16,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:16,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:16,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:16,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:16,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:16,580 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:16,580 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:16,580 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:16,582 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:16,582 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:16,582 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:16,582 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:16,582 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:16,582 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:16,582 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:16,582 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:16,582 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,582 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:16,582 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:16,583 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:16,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:16,583 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:16,584 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,584 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:16,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:16,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:16,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:16,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c768437[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:16,594 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:16,594 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,595 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,596 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:16,596 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:16,596 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,596 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:16,596 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:16,597 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:16,597 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:16,597 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:16,597 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:16,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:16,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:16,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:16,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:16,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:16,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:16,601 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:16,601 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:16,601 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:16,601 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:16,601 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:16,603 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,603 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:16,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,603 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:16,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:16,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:16,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:16,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:16,605 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:16,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,607 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:16,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,609 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:16,609 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:16,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:16,609 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:16,609 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:16,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:16,610 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,610 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,610 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,610 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,610 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:16,610 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:16,610 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:16,614 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:16,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:16,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:16,714 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:16,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:16,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:16,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:16,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:16,717 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:16,717 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:16,717 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:16,717 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:16,717 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,717 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,718 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:16,718 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:16,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:16,718 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,718 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:16,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:16,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,719 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:16,719 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:16,719 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:16,719 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:16,719 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:16,719 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:16,719 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:16,719 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:16,719 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:16,719 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:16,719 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:16,719 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:16,719 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:16,720 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:16,720 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,720 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:16,720 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:16,720 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:16,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:16,721 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:16,721 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:16,721 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:16,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:16,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:16,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d28cda7[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:16,722 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:16,722 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,722 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,723 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:16,723 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:16,723 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:16,723 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,723 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:16,723 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,723 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:16,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:16,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:16,723 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:16,724 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:16,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:16,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:16,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:16,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:16,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:16,725 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:16,725 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:16,725 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:16,725 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:16,726 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:16,727 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,727 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:16,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,727 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:16,727 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:16,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,728 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:16,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:16,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,730 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:16,730 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:16,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:16,730 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,730 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:16,730 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:16,730 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:16,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:16,730 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,730 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:16,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:16,730 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:16,730 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:16,731 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:16,731 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,760 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/9c5c8cee643c476ea289423ce15a06e6 2023-04-28 14:57:16,765 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/902c3a05af434d02b6a98cdfaf34c46d as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/902c3a05af434d02b6a98cdfaf34c46d 2023-04-28 14:57:16,770 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/902c3a05af434d02b6a98cdfaf34c46d, entries=200, sequenceid=495, filesize=14.3 K 2023-04-28 14:57:16,770 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/7dfc3ab7c3e540788967804ddeb78c11 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/7dfc3ab7c3e540788967804ddeb78c11 2023-04-28 14:57:16,775 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/7dfc3ab7c3e540788967804ddeb78c11, entries=150, sequenceid=495, filesize=11.9 K 2023-04-28 14:57:16,776 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/9c5c8cee643c476ea289423ce15a06e6 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/9c5c8cee643c476ea289423ce15a06e6 2023-04-28 14:57:16,781 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/9c5c8cee643c476ea289423ce15a06e6, entries=150, sequenceid=495, filesize=11.9 K 2023-04-28 14:57:16,782 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~394.64 KB/404112, currentSize=40.25 KB/41220 for cce278287965607462f53e33830eba7b in 901ms, sequenceid=495, compaction requested=true 2023-04-28 14:57:16,782 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:16,782 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:57:16,782 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:57:16,782 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:57:16,782 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:57:16,782 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:57:16,784 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:57:16,784 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): cce278287965607462f53e33830eba7b/B is initiating minor compaction (all files) 2023-04-28 14:57:16,784 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52002 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:57:16,784 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cce278287965607462f53e33830eba7b/B in TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:16,784 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): cce278287965607462f53e33830eba7b/A is initiating minor compaction (all files) 2023-04-28 14:57:16,784 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/9099770035b74103a5919a15d09f65a6, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/8282f02cb18d4f668fc68753ffd98afe, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/845038d42985407f8e8d99a90a93fadb, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/7dfc3ab7c3e540788967804ddeb78c11] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp, totalSize=48.4 K 2023-04-28 14:57:16,784 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cce278287965607462f53e33830eba7b/A in TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:16,784 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/b6a4a733e0954b40a7af8ef6c310f9a3, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/022f0dc399f24d7c948e0a6c81ddfab9, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/008e3f186fa248b1b15b69fd6fc3448f, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/902c3a05af434d02b6a98cdfaf34c46d] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp, totalSize=50.8 K 2023-04-28 14:57:16,784 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 9099770035b74103a5919a15d09f65a6, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1682693832824 2023-04-28 14:57:16,784 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting b6a4a733e0954b40a7af8ef6c310f9a3, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1682693832824 2023-04-28 14:57:16,785 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 8282f02cb18d4f668fc68753ffd98afe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1682693832892 2023-04-28 14:57:16,785 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 022f0dc399f24d7c948e0a6c81ddfab9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1682693832892 2023-04-28 14:57:16,785 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 845038d42985407f8e8d99a90a93fadb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1682693833547 2023-04-28 14:57:16,785 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 008e3f186fa248b1b15b69fd6fc3448f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1682693833547 2023-04-28 14:57:16,786 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 7dfc3ab7c3e540788967804ddeb78c11, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1682693834802 2023-04-28 14:57:16,788 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 902c3a05af434d02b6a98cdfaf34c46d, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1682693834802 2023-04-28 14:57:16,795 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): cce278287965607462f53e33830eba7b#B#compaction#85 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:16,796 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): cce278287965607462f53e33830eba7b#A#compaction#84 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:16,831 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:16,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:16,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:16,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:16,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:16,833 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:16,833 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:16,833 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:16,833 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:16,833 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,833 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,834 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:16,834 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:16,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:16,834 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,834 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:16,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:16,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:16,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:16,835 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:16,835 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:16,835 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:16,835 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:16,835 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:16,835 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:16,835 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:16,836 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:16,836 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:16,836 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:16,836 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cce278287965607462f53e33830eba7b 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-28 14:57:16,836 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=A 2023-04-28 14:57:16,836 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:16,836 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=B 2023-04-28 14:57:16,836 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:16,836 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=C 2023-04-28 14:57:16,836 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:16,843 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=504 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/d1f349337b2c4130b2bd9ee94c8987a5 2023-04-28 14:57:16,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on cce278287965607462f53e33830eba7b 2023-04-28 14:57:16,930 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:16,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682693836945, max timestamp=1682693836957Num uniques 859; 2023-04-28 14:57:16,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-28 14:57:16,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682693836945, max timestamp=1682693836957Num uniques 859; 2023-04-28 14:57:16,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-28 14:57:16,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682693836945, max timestamp=1682693836957Num uniques 859; 2023-04-28 14:57:16,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-28 14:57:16,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:16,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693896977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:16,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693896978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:16,982 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:16,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693896979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,008 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:17,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693897008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:17,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693897013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:17,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693897082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:17,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693897082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:17,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693897083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,208 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/91d3db0cd2eb41759c44ddfe82f5cfb7 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/91d3db0cd2eb41759c44ddfe82f5cfb7 2023-04-28 14:57:17,208 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/15541eaea7a5454fb2897a7e836d0cbc as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/15541eaea7a5454fb2897a7e836d0cbc 2023-04-28 14:57:17,219 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cce278287965607462f53e33830eba7b/A of cce278287965607462f53e33830eba7b into 91d3db0cd2eb41759c44ddfe82f5cfb7(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:17,219 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:17,219 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cce278287965607462f53e33830eba7b/B of cce278287965607462f53e33830eba7b into 15541eaea7a5454fb2897a7e836d0cbc(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:17,219 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:17,219 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b., storeName=cce278287965607462f53e33830eba7b/A, priority=12, startTime=1682693836782; duration=0sec 2023-04-28 14:57:17,219 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b., storeName=cce278287965607462f53e33830eba7b/B, priority=12, startTime=1682693836782; duration=0sec 2023-04-28 14:57:17,219 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:57:17,219 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:17,219 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:57:17,221 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:57:17,221 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): cce278287965607462f53e33830eba7b/C is initiating minor compaction (all files) 2023-04-28 14:57:17,221 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cce278287965607462f53e33830eba7b/C in TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:17,221 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/f5583a948f85434bbfee3da773638a00, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/10a2d798647142d1958a56ca1bca2b56, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/bfce4cd51e414cc9a28a84eb2275dfc1, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/9c5c8cee643c476ea289423ce15a06e6] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp, totalSize=48.4 K 2023-04-28 14:57:17,222 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting f5583a948f85434bbfee3da773638a00, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1682693832824 2023-04-28 14:57:17,222 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 10a2d798647142d1958a56ca1bca2b56, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1682693832892 2023-04-28 14:57:17,222 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting bfce4cd51e414cc9a28a84eb2275dfc1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1682693833547 2023-04-28 14:57:17,223 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 9c5c8cee643c476ea289423ce15a06e6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1682693834802 2023-04-28 14:57:17,230 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): cce278287965607462f53e33830eba7b#C#compaction#88 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:17,256 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=504 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/72aa3bd0b6a54120879ec1fad055b817 2023-04-28 14:57:17,288 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:17,289 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:17,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693897285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693897285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:17,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693897286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:17,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693897590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,593 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:17,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693897591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:17,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693897591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,657 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/ed5231d71afe41b689a78acdbf835d7e as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/ed5231d71afe41b689a78acdbf835d7e 2023-04-28 14:57:17,667 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cce278287965607462f53e33830eba7b/C of cce278287965607462f53e33830eba7b into ed5231d71afe41b689a78acdbf835d7e(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:17,668 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:17,668 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b., storeName=cce278287965607462f53e33830eba7b/C, priority=12, startTime=1682693836782; duration=0sec 2023-04-28 14:57:17,668 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:17,668 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=504 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/96b2335a1df747d58cf8e0a55fa377a0 2023-04-28 14:57:17,677 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/d1f349337b2c4130b2bd9ee94c8987a5 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/d1f349337b2c4130b2bd9ee94c8987a5 2023-04-28 14:57:17,686 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/d1f349337b2c4130b2bd9ee94c8987a5, entries=150, sequenceid=504, filesize=11.9 K 2023-04-28 14:57:17,688 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/72aa3bd0b6a54120879ec1fad055b817 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/72aa3bd0b6a54120879ec1fad055b817 2023-04-28 14:57:17,694 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/72aa3bd0b6a54120879ec1fad055b817, entries=150, sequenceid=504, filesize=11.9 K 2023-04-28 14:57:17,696 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/96b2335a1df747d58cf8e0a55fa377a0 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/96b2335a1df747d58cf8e0a55fa377a0 2023-04-28 14:57:17,710 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/96b2335a1df747d58cf8e0a55fa377a0, entries=150, sequenceid=504, filesize=11.9 K 2023-04-28 14:57:17,711 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=147.60 KB/151140 for cce278287965607462f53e33830eba7b in 875ms, sequenceid=504, compaction requested=false 2023-04-28 14:57:17,711 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:17,711 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:17,711 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:17,711 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:17,711 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,712 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:17,712 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:17,714 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:17,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:17,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:17,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:17,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:17,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:17,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:17,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@676a39a9[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:17,716 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:17,716 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:17,717 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:17,717 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:17,717 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:17,717 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:17,717 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:17,717 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:17,717 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:17,718 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,718 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:17,719 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:17,719 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:17,719 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,719 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:17,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:17,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:17,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:17,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:17,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:17,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:17,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:17,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:17,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:17,724 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:17,724 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:17,724 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:17,724 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:17,724 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:17,727 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:17,727 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:17,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:17,727 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:17,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:17,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:17,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:17,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:17,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:17,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:17,728 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:17,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:17,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:17,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:17,729 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:17,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:17,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:17,731 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:17,731 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:17,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:17,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:17,732 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:17,732 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:17,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:17,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:17,732 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,732 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:17,732 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:17,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:17,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:17,732 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:17,732 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,732 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:17,732 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:17,733 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:17,733 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:17,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:17,833 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:17,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:17,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:17,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:17,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:17,842 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:17,842 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:17,842 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:17,842 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:17,843 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:17,843 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:17,844 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:17,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:17,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:17,844 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:17,844 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,844 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:17,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:17,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:17,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:17,845 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:17,854 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:17,854 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:17,854 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:17,854 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:17,854 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:17,863 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:17,863 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:17,865 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:17,866 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:17,866 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cce278287965607462f53e33830eba7b 3/3 column families, dataSize=147.60 KB heapSize=406.08 KB 2023-04-28 14:57:17,866 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=A 2023-04-28 14:57:17,866 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:17,866 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=B 2023-04-28 14:57:17,866 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:17,866 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=C 2023-04-28 14:57:17,866 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:17,896 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=542 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/9374ccd6829f434d8e006a67cb225263 2023-04-28 14:57:17,913 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=542 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/f6a158492feb445d8e13007c5100532f 2023-04-28 14:57:17,929 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=542 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/d583e095bb10461a816d1b64aab00740 2023-04-28 14:57:17,936 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/9374ccd6829f434d8e006a67cb225263 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/9374ccd6829f434d8e006a67cb225263 2023-04-28 14:57:17,941 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/9374ccd6829f434d8e006a67cb225263, entries=150, sequenceid=542, filesize=11.9 K 2023-04-28 14:57:17,943 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/f6a158492feb445d8e13007c5100532f as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/f6a158492feb445d8e13007c5100532f 2023-04-28 14:57:17,949 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/f6a158492feb445d8e13007c5100532f, entries=150, sequenceid=542, filesize=11.9 K 2023-04-28 14:57:17,950 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/d583e095bb10461a816d1b64aab00740 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/d583e095bb10461a816d1b64aab00740 2023-04-28 14:57:17,955 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/d583e095bb10461a816d1b64aab00740, entries=150, sequenceid=542, filesize=11.9 K 2023-04-28 14:57:17,955 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~406.03 KB/415776, currentSize=0 B/0 for cce278287965607462f53e33830eba7b in 89ms, sequenceid=542, compaction requested=true 2023-04-28 14:57:17,955 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:17,956 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:17,956 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:17,956 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:17,956 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,956 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:17,956 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:17,957 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:17,957 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:17,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:17,957 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:17,957 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:17,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:17,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:17,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:17,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:17,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:17,962 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:17,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d6898d1[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:17,962 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:17,963 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:17,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:17,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:17,963 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,963 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:17,963 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:17,963 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:17,963 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:17,964 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:17,964 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,964 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:17,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:17,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:17,964 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:17,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:17,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:17,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:17,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:17,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:17,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:17,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:17,967 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:17,967 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:17,967 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:17,967 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:17,967 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:17,969 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:17,969 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:17,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:17,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:17,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:17,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:17,969 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:17,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:17,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:17,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:17,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:17,970 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:17,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:17,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:17,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:17,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,971 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:17,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:17,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,974 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,974 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:17,974 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:17,974 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:17,975 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:17,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:17,975 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:17,975 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:17,975 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:17,975 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:17,975 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:17,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:17,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:17,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:17,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:17,975 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:17,975 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:17,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:17,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:18,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on cce278287965607462f53e33830eba7b 2023-04-28 14:57:18,027 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cce278287965607462f53e33830eba7b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 14:57:18,028 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=A 2023-04-28 14:57:18,028 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:18,028 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=B 2023-04-28 14:57:18,028 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:18,028 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=C 2023-04-28 14:57:18,028 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:18,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682693838045, max timestamp=1682693838059Num uniques 859; 2023-04-28 14:57:18,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 14:57:18,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682693838045, max timestamp=1682693838059Num uniques 859; 2023-04-28 14:57:18,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 14:57:18,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682693838045, max timestamp=1682693838059Num uniques 859; 2023-04-28 14:57:18,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 14:57:18,075 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:18,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:18,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:18,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:18,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:18,078 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:18,078 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:18,078 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:18,078 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:18,079 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:18,079 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:18,079 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,080 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:18,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:18,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:18,080 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,080 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:18,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:18,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:18,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:18,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:18,081 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:18,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:18,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:18,081 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:18,081 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:18,081 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:18,081 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:18,081 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:18,081 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:18,081 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:18,081 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:18,082 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:18,082 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:18,082 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:18,082 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,082 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:18,082 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:18,084 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:18,084 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:18,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:18,084 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:18,084 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:18,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:18,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:18,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:18,086 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:18,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25f7fe77[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:18,086 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:18,086 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:18,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:18,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:18,087 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:18,087 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:18,087 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:18,087 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,087 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:18,088 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,088 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:18,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:18,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:18,088 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:18,088 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:18,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:18,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:18,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:18,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:18,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:18,090 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:18,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:18,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693898087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,091 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:18,091 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:18,091 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:18,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:18,091 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:18,091 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:18,091 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:18,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693898087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,092 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:18,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:18,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:18,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:18,093 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:18,093 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:18,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:18,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:18,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:18,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:18,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:18,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:18,094 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:18,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:18,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:18,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,095 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:18,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:18,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:18,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693898094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:18,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693898097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,097 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,097 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:18,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:18,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:18,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:18,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693898097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,097 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:18,097 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:18,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:18,097 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:18,098 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:18,098 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,098 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:18,098 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:18,098 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:18,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:18,098 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:18,098 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:18,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:18,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:18,193 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:18,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693898192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:18,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693898192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,198 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:18,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:18,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:18,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:18,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:18,202 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:18,202 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:18,202 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:18,202 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:18,203 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:18,203 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:18,203 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,203 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:18,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:18,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:18,204 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,204 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:18,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:18,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:18,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:18,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:18,204 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:18,205 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:18,205 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:18,205 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:18,205 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:18,205 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:18,206 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:18,206 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:18,206 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:18,206 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. started... 2023-04-28 14:57:18,206 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. as already flushing 2023-04-28 14:57:18,206 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:18,206 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:18,206 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:18,206 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,206 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:18,206 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:18,207 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:18,207 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:18,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:18,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:18,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:18,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:18,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:18,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:18,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:18,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:18,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ee95a9b[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:18,209 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:18,209 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:18,210 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:18,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:18,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:18,210 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:18,210 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:18,210 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:18,210 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,210 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:18,211 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:18,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:18,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:18,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:18,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:18,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:18,214 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:18,214 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:18,214 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:18,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:18,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:18,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:18,219 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:18,219 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:18,219 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:18,219 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:18,219 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:18,221 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:18,221 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:18,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:18,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:18,221 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:18,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:18,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:18,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:18,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:18,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:18,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:18,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:18,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:18,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:18,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:18,224 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:18,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:18,225 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,228 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:18,228 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:18,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:18,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:18,228 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,228 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:18,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:18,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:18,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:18,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:18,228 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:18,228 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:18,228 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:18,228 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:18,229 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,229 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:18,229 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:18,229 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:18,282 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3fed686f to 127.0.0.1:61762 2023-04-28 14:57:18,282 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:57:18,286 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e044e22 to 127.0.0.1:61762 2023-04-28 14:57:18,286 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:57:18,287 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x18f7ec13 to 127.0.0.1:61762 2023-04-28 14:57:18,287 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:57:18,289 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ab9467e to 127.0.0.1:61762 2023-04-28 14:57:18,289 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:57:18,328 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:18,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:18,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:18,396 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:18,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693898396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693898396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,437 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=552 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/09a2af3b12e54c44b6e6976e355e4b37 2023-04-28 14:57:18,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:18,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:18,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693898698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693898698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:18,849 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=552 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/31860cfdbf0e4de0b37d6e6663545cec 2023-04-28 14:57:19,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:19,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53134 deadline: 1682693899099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:19,101 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:19,101 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:19,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53108 deadline: 1682693899101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:19,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53114 deadline: 1682693899101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:19,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:19,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53136 deadline: 1682693899203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:19,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:19,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:53138 deadline: 1682693899205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:19,261 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=552 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/30e275e6aaed4428a5ff38abee7767ef 2023-04-28 14:57:19,266 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/09a2af3b12e54c44b6e6976e355e4b37 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/09a2af3b12e54c44b6e6976e355e4b37 2023-04-28 14:57:19,272 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/09a2af3b12e54c44b6e6976e355e4b37, entries=200, sequenceid=552, filesize=14.3 K 2023-04-28 14:57:19,273 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/31860cfdbf0e4de0b37d6e6663545cec as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/31860cfdbf0e4de0b37d6e6663545cec 2023-04-28 14:57:19,278 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/31860cfdbf0e4de0b37d6e6663545cec, entries=150, sequenceid=552, filesize=11.9 K 2023-04-28 14:57:19,279 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/30e275e6aaed4428a5ff38abee7767ef as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/30e275e6aaed4428a5ff38abee7767ef 2023-04-28 14:57:19,285 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/30e275e6aaed4428a5ff38abee7767ef, entries=150, sequenceid=552, filesize=11.9 K 2023-04-28 14:57:19,286 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for cce278287965607462f53e33830eba7b in 1259ms, sequenceid=552, compaction requested=true 2023-04-28 14:57:19,286 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:19,286 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:57:19,286 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:57:19,286 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:57:19,286 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:57:19,286 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:57:19,288 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49698 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:57:19,288 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52138 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:57:19,288 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): cce278287965607462f53e33830eba7b/B is initiating minor compaction (all files) 2023-04-28 14:57:19,288 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): cce278287965607462f53e33830eba7b/A is initiating minor compaction (all files) 2023-04-28 14:57:19,288 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cce278287965607462f53e33830eba7b/B in TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:19,288 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cce278287965607462f53e33830eba7b/A in TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:19,288 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/91d3db0cd2eb41759c44ddfe82f5cfb7, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/d1f349337b2c4130b2bd9ee94c8987a5, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/9374ccd6829f434d8e006a67cb225263, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/09a2af3b12e54c44b6e6976e355e4b37] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp, totalSize=50.9 K 2023-04-28 14:57:19,288 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/15541eaea7a5454fb2897a7e836d0cbc, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/72aa3bd0b6a54120879ec1fad055b817, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/f6a158492feb445d8e13007c5100532f, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/31860cfdbf0e4de0b37d6e6663545cec] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp, totalSize=48.5 K 2023-04-28 14:57:19,288 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 91d3db0cd2eb41759c44ddfe82f5cfb7, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1682693834802 2023-04-28 14:57:19,288 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 15541eaea7a5454fb2897a7e836d0cbc, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1682693834802 2023-04-28 14:57:19,289 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting d1f349337b2c4130b2bd9ee94c8987a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1682693835889 2023-04-28 14:57:19,289 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 72aa3bd0b6a54120879ec1fad055b817, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1682693835889 2023-04-28 14:57:19,289 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 9374ccd6829f434d8e006a67cb225263, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=542, earliestPutTs=1682693836970 2023-04-28 14:57:19,289 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting f6a158492feb445d8e13007c5100532f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=542, earliestPutTs=1682693836970 2023-04-28 14:57:19,289 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 09a2af3b12e54c44b6e6976e355e4b37, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=552, earliestPutTs=1682693838016 2023-04-28 14:57:19,289 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 31860cfdbf0e4de0b37d6e6663545cec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=552, earliestPutTs=1682693838016 2023-04-28 14:57:19,299 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): cce278287965607462f53e33830eba7b#A#compaction#96 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:19,301 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): cce278287965607462f53e33830eba7b#B#compaction#97 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:19,715 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/65f0be0b151340a9a56f79a107f32ff2 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/65f0be0b151340a9a56f79a107f32ff2 2023-04-28 14:57:19,715 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/39fc03dbfbc14a55af36c57a747e6195 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/39fc03dbfbc14a55af36c57a747e6195 2023-04-28 14:57:19,720 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cce278287965607462f53e33830eba7b/A of cce278287965607462f53e33830eba7b into 65f0be0b151340a9a56f79a107f32ff2(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:19,720 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cce278287965607462f53e33830eba7b/B of cce278287965607462f53e33830eba7b into 39fc03dbfbc14a55af36c57a747e6195(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:19,720 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:19,720 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:19,721 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b., storeName=cce278287965607462f53e33830eba7b/A, priority=12, startTime=1682693839286; duration=0sec 2023-04-28 14:57:19,721 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b., storeName=cce278287965607462f53e33830eba7b/B, priority=12, startTime=1682693839286; duration=0sec 2023-04-28 14:57:19,721 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:57:19,721 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:19,721 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:57:19,722 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49698 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:57:19,722 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): cce278287965607462f53e33830eba7b/C is initiating minor compaction (all files) 2023-04-28 14:57:19,722 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cce278287965607462f53e33830eba7b/C in TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:19,722 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/ed5231d71afe41b689a78acdbf835d7e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/96b2335a1df747d58cf8e0a55fa377a0, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/d583e095bb10461a816d1b64aab00740, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/30e275e6aaed4428a5ff38abee7767ef] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp, totalSize=48.5 K 2023-04-28 14:57:19,723 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting ed5231d71afe41b689a78acdbf835d7e, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1682693834802 2023-04-28 14:57:19,723 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 96b2335a1df747d58cf8e0a55fa377a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1682693835889 2023-04-28 14:57:19,723 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting d583e095bb10461a816d1b64aab00740, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=542, earliestPutTs=1682693836970 2023-04-28 14:57:19,724 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 30e275e6aaed4428a5ff38abee7767ef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=552, earliestPutTs=1682693838016 2023-04-28 14:57:19,731 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): cce278287965607462f53e33830eba7b#C#compaction#98 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:20,142 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/0beaef79f290476f86208b332c07c948 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/0beaef79f290476f86208b332c07c948 2023-04-28 14:57:20,148 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cce278287965607462f53e33830eba7b/C of cce278287965607462f53e33830eba7b into 0beaef79f290476f86208b332c07c948(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:20,148 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:20,148 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b., storeName=cce278287965607462f53e33830eba7b/C, priority=12, startTime=1682693839286; duration=0sec 2023-04-28 14:57:20,148 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:20,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on cce278287965607462f53e33830eba7b 2023-04-28 14:57:20,210 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cce278287965607462f53e33830eba7b 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-28 14:57:20,210 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x77443fb0 to 127.0.0.1:61762 2023-04-28 14:57:20,211 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=A 2023-04-28 14:57:20,211 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:57:20,211 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:20,211 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=B 2023-04-28 14:57:20,211 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:20,211 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=C 2023-04-28 14:57:20,212 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:20,216 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x45864842 to 127.0.0.1:61762 2023-04-28 14:57:20,216 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:57:20,623 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=591 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/18f7133fa5b445fd8241cb7d5370b487 2023-04-28 14:57:21,038 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=591 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/4f6636ca9ac84bf5ad90b20abf8364dd 2023-04-28 14:57:21,111 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ecabb06 to 127.0.0.1:61762 2023-04-28 14:57:21,111 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:57:21,114 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66ca212c to 127.0.0.1:61762 2023-04-28 14:57:21,114 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:57:21,121 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4654132d to 127.0.0.1:61762 2023-04-28 14:57:21,121 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:57:21,121 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-28 14:57:21,121 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(394): wrote 94 2023-04-28 14:57:21,121 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(394): wrote 94 2023-04-28 14:57:21,121 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(394): wrote 90 2023-04-28 14:57:21,122 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(394): wrote 114 2023-04-28 14:57:21,122 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(394): wrote 100 2023-04-28 14:57:21,122 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-28 14:57:21,122 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(398): read 3970 2023-04-28 14:57:21,122 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(398): read 3876 2023-04-28 14:57:21,122 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-28 14:57:21,122 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(402): scanned 1745 2023-04-28 14:57:21,122 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(403): verified 5231 rows 2023-04-28 14:57:21,122 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(402): scanned 1746 2023-04-28 14:57:21,122 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(403): verified 5234 rows 2023-04-28 14:57:21,122 INFO [Listener at localhost.localdomain/40193] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-28 14:57:21,123 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x490ff358 to 127.0.0.1:61762 2023-04-28 14:57:21,123 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:57:21,128 INFO [Listener at localhost.localdomain/40193] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-28 14:57:21,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.HMaster$11(2418): Client=jenkins//136.243.18.41 disable TestAcidGuarantees 2023-04-28 14:57:21,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-28 14:57:21,143 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682693841142"}]},"ts":"1682693841142"} 2023-04-28 14:57:21,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-28 14:57:21,144 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-28 14:57:21,145 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-28 14:57:21,146 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=cce278287965607462f53e33830eba7b, UNASSIGN}] 2023-04-28 14:57:21,147 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=cce278287965607462f53e33830eba7b, UNASSIGN 2023-04-28 14:57:21,148 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=cce278287965607462f53e33830eba7b, regionState=CLOSING, regionLocation=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:21,148 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693841148"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682693841148"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682693841148"}]},"ts":"1682693841148"} 2023-04-28 14:57:21,149 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482}] 2023-04-28 14:57:21,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-28 14:57:21,310 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(109): Close cce278287965607462f53e33830eba7b 2023-04-28 14:57:21,311 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1604): Closing cce278287965607462f53e33830eba7b, disabling compactions & flushes 2023-04-28 14:57:21,311 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:21,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-28 14:57:21,450 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=591 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/94cab253e34745aea2492787fb63e122 2023-04-28 14:57:21,462 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/18f7133fa5b445fd8241cb7d5370b487 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/18f7133fa5b445fd8241cb7d5370b487 2023-04-28 14:57:21,469 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/18f7133fa5b445fd8241cb7d5370b487, entries=150, sequenceid=591, filesize=11.9 K 2023-04-28 14:57:21,470 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/4f6636ca9ac84bf5ad90b20abf8364dd as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/4f6636ca9ac84bf5ad90b20abf8364dd 2023-04-28 14:57:21,477 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/4f6636ca9ac84bf5ad90b20abf8364dd, entries=150, sequenceid=591, filesize=11.9 K 2023-04-28 14:57:21,479 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/94cab253e34745aea2492787fb63e122 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/94cab253e34745aea2492787fb63e122 2023-04-28 14:57:21,485 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/94cab253e34745aea2492787fb63e122, entries=150, sequenceid=591, filesize=11.9 K 2023-04-28 14:57:21,486 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=26.84 KB/27480 for cce278287965607462f53e33830eba7b in 1276ms, sequenceid=591, compaction requested=false 2023-04-28 14:57:21,486 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:21,486 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:21,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:21,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. after waiting 0 ms 2023-04-28 14:57:21,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:21,487 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2745): Flushing cce278287965607462f53e33830eba7b 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-28 14:57:21,487 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=A 2023-04-28 14:57:21,487 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:21,487 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=B 2023-04-28 14:57:21,487 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:21,487 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cce278287965607462f53e33830eba7b, store=C 2023-04-28 14:57:21,487 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:21,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-28 14:57:21,897 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=598 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/ce0f314a00984e43b5d87292ae0aca25 2023-04-28 14:57:22,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-28 14:57:22,315 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=598 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/ef2ec2d595694272843fe0a37ae780db 2023-04-28 14:57:22,733 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=598 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/4e21da5813514ab180044f23b9985f6d 2023-04-28 14:57:22,747 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/A/ce0f314a00984e43b5d87292ae0aca25 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/ce0f314a00984e43b5d87292ae0aca25 2023-04-28 14:57:22,753 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/ce0f314a00984e43b5d87292ae0aca25, entries=100, sequenceid=598, filesize=9.5 K 2023-04-28 14:57:22,754 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/B/ef2ec2d595694272843fe0a37ae780db as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/ef2ec2d595694272843fe0a37ae780db 2023-04-28 14:57:22,759 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/ef2ec2d595694272843fe0a37ae780db, entries=100, sequenceid=598, filesize=9.5 K 2023-04-28 14:57:22,760 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/.tmp/C/4e21da5813514ab180044f23b9985f6d as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/4e21da5813514ab180044f23b9985f6d 2023-04-28 14:57:22,765 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/4e21da5813514ab180044f23b9985f6d, entries=100, sequenceid=598, filesize=9.5 K 2023-04-28 14:57:22,766 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for cce278287965607462f53e33830eba7b in 1279ms, sequenceid=598, compaction requested=true 2023-04-28 14:57:22,786 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/4fdf12af22864055807f1492dadcd192, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/1bfeecfa7d8b4c41aa8b36b1ce397501, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/6da170faf0be4c0fa7fbda6902573263, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/b3b3655f41e44f1986c99252981c1a8b, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/2130444c2f154bd7af647c88371269cd, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/24c860bb48ce40c9b305b336862cb493, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/ab2de3b398e044c1a460a7351930ff51, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/d592811967df4f9b9007ece20d8b86af, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/fee7e46c741549b79fe1822b462797ce, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/32a28c1ecb7d4e4cbe536de007a3739c, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/03d005ce43414181aced3fb39fe09228, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/c6e03d9edda04c6ca8d23f5fd63c1315, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/7b03223476a24ded87e6f85f1183b80c, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/27f76f301fbc4f52adcf1ac9dc93cdb2, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/e5302fba6efa4bdc9e8e2d4f9d4d4a0a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/83455c337ff4444bb4b2764de10b95fb, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/92e9978209b4410bb869cde95a40ce9e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/df9fb7c8121c498e89c5891ea713f5bd, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/a794d60fddae4b528f6b1aae45a18639, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/ba04af465b314451aa1bf44fcce90f88, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/0549671a248844f581fb4725194c32f1, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/3f6d55ba5f9e49b98c006f0828067516, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/12855529b6cb4d938d447a48680a4ee5, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/e485cbd54be0476999aba2cd9e16881b, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/b6a4a733e0954b40a7af8ef6c310f9a3, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/022f0dc399f24d7c948e0a6c81ddfab9, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/008e3f186fa248b1b15b69fd6fc3448f, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/902c3a05af434d02b6a98cdfaf34c46d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/91d3db0cd2eb41759c44ddfe82f5cfb7, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/d1f349337b2c4130b2bd9ee94c8987a5, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/9374ccd6829f434d8e006a67cb225263, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/09a2af3b12e54c44b6e6976e355e4b37] to archive 2023-04-28 14:57:22,787 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 14:57:22,793 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/4fdf12af22864055807f1492dadcd192 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/4fdf12af22864055807f1492dadcd192 2023-04-28 14:57:22,794 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/1bfeecfa7d8b4c41aa8b36b1ce397501 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/1bfeecfa7d8b4c41aa8b36b1ce397501 2023-04-28 14:57:22,796 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/6da170faf0be4c0fa7fbda6902573263 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/6da170faf0be4c0fa7fbda6902573263 2023-04-28 14:57:22,797 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/b3b3655f41e44f1986c99252981c1a8b to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/b3b3655f41e44f1986c99252981c1a8b 2023-04-28 14:57:22,799 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/2130444c2f154bd7af647c88371269cd to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/2130444c2f154bd7af647c88371269cd 2023-04-28 14:57:22,801 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/24c860bb48ce40c9b305b336862cb493 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/24c860bb48ce40c9b305b336862cb493 2023-04-28 14:57:22,803 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/ab2de3b398e044c1a460a7351930ff51 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/ab2de3b398e044c1a460a7351930ff51 2023-04-28 14:57:22,804 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/d592811967df4f9b9007ece20d8b86af to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/d592811967df4f9b9007ece20d8b86af 2023-04-28 14:57:22,806 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/fee7e46c741549b79fe1822b462797ce to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/fee7e46c741549b79fe1822b462797ce 2023-04-28 14:57:22,807 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/32a28c1ecb7d4e4cbe536de007a3739c to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/32a28c1ecb7d4e4cbe536de007a3739c 2023-04-28 14:57:22,808 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/03d005ce43414181aced3fb39fe09228 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/03d005ce43414181aced3fb39fe09228 2023-04-28 14:57:22,810 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/c6e03d9edda04c6ca8d23f5fd63c1315 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/c6e03d9edda04c6ca8d23f5fd63c1315 2023-04-28 14:57:22,811 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/7b03223476a24ded87e6f85f1183b80c to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/7b03223476a24ded87e6f85f1183b80c 2023-04-28 14:57:22,813 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/27f76f301fbc4f52adcf1ac9dc93cdb2 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/27f76f301fbc4f52adcf1ac9dc93cdb2 2023-04-28 14:57:22,814 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/e5302fba6efa4bdc9e8e2d4f9d4d4a0a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/e5302fba6efa4bdc9e8e2d4f9d4d4a0a 2023-04-28 14:57:22,816 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/83455c337ff4444bb4b2764de10b95fb to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/83455c337ff4444bb4b2764de10b95fb 2023-04-28 14:57:22,817 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/92e9978209b4410bb869cde95a40ce9e to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/92e9978209b4410bb869cde95a40ce9e 2023-04-28 14:57:22,818 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/df9fb7c8121c498e89c5891ea713f5bd to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/df9fb7c8121c498e89c5891ea713f5bd 2023-04-28 14:57:22,820 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/a794d60fddae4b528f6b1aae45a18639 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/a794d60fddae4b528f6b1aae45a18639 2023-04-28 14:57:22,821 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/ba04af465b314451aa1bf44fcce90f88 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/ba04af465b314451aa1bf44fcce90f88 2023-04-28 14:57:22,822 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/0549671a248844f581fb4725194c32f1 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/0549671a248844f581fb4725194c32f1 2023-04-28 14:57:22,823 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/3f6d55ba5f9e49b98c006f0828067516 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/3f6d55ba5f9e49b98c006f0828067516 2023-04-28 14:57:22,824 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/12855529b6cb4d938d447a48680a4ee5 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/12855529b6cb4d938d447a48680a4ee5 2023-04-28 14:57:22,826 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/e485cbd54be0476999aba2cd9e16881b to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/e485cbd54be0476999aba2cd9e16881b 2023-04-28 14:57:22,827 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/b6a4a733e0954b40a7af8ef6c310f9a3 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/b6a4a733e0954b40a7af8ef6c310f9a3 2023-04-28 14:57:22,828 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/022f0dc399f24d7c948e0a6c81ddfab9 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/022f0dc399f24d7c948e0a6c81ddfab9 2023-04-28 14:57:22,829 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/008e3f186fa248b1b15b69fd6fc3448f to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/008e3f186fa248b1b15b69fd6fc3448f 2023-04-28 14:57:22,830 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/902c3a05af434d02b6a98cdfaf34c46d to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/902c3a05af434d02b6a98cdfaf34c46d 2023-04-28 14:57:22,832 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/91d3db0cd2eb41759c44ddfe82f5cfb7 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/91d3db0cd2eb41759c44ddfe82f5cfb7 2023-04-28 14:57:22,833 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/d1f349337b2c4130b2bd9ee94c8987a5 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/d1f349337b2c4130b2bd9ee94c8987a5 2023-04-28 14:57:22,834 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/9374ccd6829f434d8e006a67cb225263 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/9374ccd6829f434d8e006a67cb225263 2023-04-28 14:57:22,835 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/09a2af3b12e54c44b6e6976e355e4b37 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/09a2af3b12e54c44b6e6976e355e4b37 2023-04-28 14:57:22,858 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/59f60be6f15c4048bbb98871bbf32bd5, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/2f7c14d7b8714d7599f10a54db4afaaf, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/cdd79dacfa474ca98d4a23c45794bc7c, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/5b593c12ec764220a541d668e587e298, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/462d01412a0a4aa8877e9fe946b49fed, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/a446af92ed254fcf86edb27e2e59d64d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/473b1abc37f649598dd6753d62fbafe6, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/af40aab89bdc4ad7843cb8f4d365aa53, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/22f93f78dba547d6bdb54e7c711b12f8, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/fad4a3db93b44a049708afcaf4c806e3, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/96b260db18aa4cfb8c1c311511caaff7, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/80363376c29448ce96cd9c00c6382999, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/efef38f3ded34c37aabf653b8bd2714f, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/cb01a112c018439e90757ec6069a94ad, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/a790f9dc2ff04f20ad2ec4a699d34f1a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/f498f8c613cb4ce1810a37526b69b17a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/b708905bbf4849a6a39bfae2827e9a8b, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/a7396c0b9cc946efb904be055e45f5bf, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/e2abdb9125fc480e945bf7de5a09479b, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/5c428c352a35496ba5066c471891218e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/d19122c1c49c4547948113d298978a21, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/f43290f08a5c426db452ea946cad317a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/6c02d9c58fdd4ff2a8c11d0ec8829970, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/9099770035b74103a5919a15d09f65a6, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/f5c42110a008444da4d6111237363e3d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/8282f02cb18d4f668fc68753ffd98afe, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/845038d42985407f8e8d99a90a93fadb, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/15541eaea7a5454fb2897a7e836d0cbc, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/7dfc3ab7c3e540788967804ddeb78c11, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/72aa3bd0b6a54120879ec1fad055b817, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/f6a158492feb445d8e13007c5100532f, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/31860cfdbf0e4de0b37d6e6663545cec] to archive 2023-04-28 14:57:22,859 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 14:57:22,861 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/59f60be6f15c4048bbb98871bbf32bd5 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/59f60be6f15c4048bbb98871bbf32bd5 2023-04-28 14:57:22,862 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/2f7c14d7b8714d7599f10a54db4afaaf to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/2f7c14d7b8714d7599f10a54db4afaaf 2023-04-28 14:57:22,863 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/cdd79dacfa474ca98d4a23c45794bc7c to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/cdd79dacfa474ca98d4a23c45794bc7c 2023-04-28 14:57:22,864 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/5b593c12ec764220a541d668e587e298 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/5b593c12ec764220a541d668e587e298 2023-04-28 14:57:22,865 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/462d01412a0a4aa8877e9fe946b49fed to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/462d01412a0a4aa8877e9fe946b49fed 2023-04-28 14:57:22,867 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/a446af92ed254fcf86edb27e2e59d64d to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/a446af92ed254fcf86edb27e2e59d64d 2023-04-28 14:57:22,868 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/473b1abc37f649598dd6753d62fbafe6 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/473b1abc37f649598dd6753d62fbafe6 2023-04-28 14:57:22,869 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/af40aab89bdc4ad7843cb8f4d365aa53 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/af40aab89bdc4ad7843cb8f4d365aa53 2023-04-28 14:57:22,871 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/22f93f78dba547d6bdb54e7c711b12f8 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/22f93f78dba547d6bdb54e7c711b12f8 2023-04-28 14:57:22,872 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/fad4a3db93b44a049708afcaf4c806e3 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/fad4a3db93b44a049708afcaf4c806e3 2023-04-28 14:57:22,873 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/96b260db18aa4cfb8c1c311511caaff7 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/96b260db18aa4cfb8c1c311511caaff7 2023-04-28 14:57:22,874 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/80363376c29448ce96cd9c00c6382999 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/80363376c29448ce96cd9c00c6382999 2023-04-28 14:57:22,876 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/efef38f3ded34c37aabf653b8bd2714f to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/efef38f3ded34c37aabf653b8bd2714f 2023-04-28 14:57:22,877 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/cb01a112c018439e90757ec6069a94ad to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/cb01a112c018439e90757ec6069a94ad 2023-04-28 14:57:22,878 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/a790f9dc2ff04f20ad2ec4a699d34f1a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/a790f9dc2ff04f20ad2ec4a699d34f1a 2023-04-28 14:57:22,879 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/f498f8c613cb4ce1810a37526b69b17a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/f498f8c613cb4ce1810a37526b69b17a 2023-04-28 14:57:22,880 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/b708905bbf4849a6a39bfae2827e9a8b to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/b708905bbf4849a6a39bfae2827e9a8b 2023-04-28 14:57:22,881 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/a7396c0b9cc946efb904be055e45f5bf to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/a7396c0b9cc946efb904be055e45f5bf 2023-04-28 14:57:22,882 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/e2abdb9125fc480e945bf7de5a09479b to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/e2abdb9125fc480e945bf7de5a09479b 2023-04-28 14:57:22,883 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/5c428c352a35496ba5066c471891218e to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/5c428c352a35496ba5066c471891218e 2023-04-28 14:57:22,885 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/d19122c1c49c4547948113d298978a21 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/d19122c1c49c4547948113d298978a21 2023-04-28 14:57:22,886 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/f43290f08a5c426db452ea946cad317a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/f43290f08a5c426db452ea946cad317a 2023-04-28 14:57:22,887 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/6c02d9c58fdd4ff2a8c11d0ec8829970 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/6c02d9c58fdd4ff2a8c11d0ec8829970 2023-04-28 14:57:22,888 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/9099770035b74103a5919a15d09f65a6 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/9099770035b74103a5919a15d09f65a6 2023-04-28 14:57:22,889 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/f5c42110a008444da4d6111237363e3d to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/f5c42110a008444da4d6111237363e3d 2023-04-28 14:57:22,890 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/8282f02cb18d4f668fc68753ffd98afe to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/8282f02cb18d4f668fc68753ffd98afe 2023-04-28 14:57:22,891 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/845038d42985407f8e8d99a90a93fadb to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/845038d42985407f8e8d99a90a93fadb 2023-04-28 14:57:22,892 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/15541eaea7a5454fb2897a7e836d0cbc to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/15541eaea7a5454fb2897a7e836d0cbc 2023-04-28 14:57:22,893 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/7dfc3ab7c3e540788967804ddeb78c11 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/7dfc3ab7c3e540788967804ddeb78c11 2023-04-28 14:57:22,894 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/72aa3bd0b6a54120879ec1fad055b817 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/72aa3bd0b6a54120879ec1fad055b817 2023-04-28 14:57:22,895 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/f6a158492feb445d8e13007c5100532f to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/f6a158492feb445d8e13007c5100532f 2023-04-28 14:57:22,896 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/31860cfdbf0e4de0b37d6e6663545cec to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/31860cfdbf0e4de0b37d6e6663545cec 2023-04-28 14:57:22,900 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/5de7ffe8dd4a42328e330523983e791f, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/9c64a928e5ed41788ef3a4e1737c710e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/8f55aa855e0949a88d22561c2a4bc65b, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/0d7c7c781ab4410bb36c6f27dbbc321a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/2e44bde400f04fa281c4d11d0e9c90b9, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/6b183ee10b03416886c54daacdd9f5a7, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/61579ac381414e929dfca6576e6704ae, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/9a98a7e3198641b3bb8172ae10014ae3, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/fcded43779504c198cba0fae83a96579, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/c5a9755f97e640e69f4036fcad499e28, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/1d252876fadb4e09aff9b1f8ace7cf56, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/f1192348fc884b2283ab04f539b31400, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/956994df71a0433bb625328da3dd5142, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/18c8c5124a7b436e8214367177e7761e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/088f581f7ce84f5a90640eac49332747, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/ba43580190f5438b98af185d3ef061c5, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/737e68e8f0cf40cf8ba35195a357f97c, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/9002a2ab62644ca08949a617d26ea42b, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/ea4ae34ca6cc4fcf861d674f345e2685, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/564671477c2842e0ac5ddfa9c9fdca0e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/09ce35b392494a8290ecad18442c98c0, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/a415c8043f54483bba57b839146993bd, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/af3ea4cb90a2466aa4d09f1ed5aba1b7, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/f5583a948f85434bbfee3da773638a00, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/5120d6cfc9024c939c163044613b68ec, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/10a2d798647142d1958a56ca1bca2b56, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/bfce4cd51e414cc9a28a84eb2275dfc1, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/ed5231d71afe41b689a78acdbf835d7e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/9c5c8cee643c476ea289423ce15a06e6, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/96b2335a1df747d58cf8e0a55fa377a0, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/d583e095bb10461a816d1b64aab00740, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/30e275e6aaed4428a5ff38abee7767ef] to archive 2023-04-28 14:57:22,901 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 14:57:22,903 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/5de7ffe8dd4a42328e330523983e791f to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/5de7ffe8dd4a42328e330523983e791f 2023-04-28 14:57:22,904 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/9c64a928e5ed41788ef3a4e1737c710e to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/9c64a928e5ed41788ef3a4e1737c710e 2023-04-28 14:57:22,905 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/8f55aa855e0949a88d22561c2a4bc65b to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/8f55aa855e0949a88d22561c2a4bc65b 2023-04-28 14:57:22,906 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/0d7c7c781ab4410bb36c6f27dbbc321a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/0d7c7c781ab4410bb36c6f27dbbc321a 2023-04-28 14:57:22,907 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/2e44bde400f04fa281c4d11d0e9c90b9 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/2e44bde400f04fa281c4d11d0e9c90b9 2023-04-28 14:57:22,908 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/6b183ee10b03416886c54daacdd9f5a7 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/6b183ee10b03416886c54daacdd9f5a7 2023-04-28 14:57:22,909 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/61579ac381414e929dfca6576e6704ae to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/61579ac381414e929dfca6576e6704ae 2023-04-28 14:57:22,910 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/9a98a7e3198641b3bb8172ae10014ae3 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/9a98a7e3198641b3bb8172ae10014ae3 2023-04-28 14:57:22,911 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/fcded43779504c198cba0fae83a96579 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/fcded43779504c198cba0fae83a96579 2023-04-28 14:57:22,912 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/c5a9755f97e640e69f4036fcad499e28 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/c5a9755f97e640e69f4036fcad499e28 2023-04-28 14:57:22,913 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/1d252876fadb4e09aff9b1f8ace7cf56 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/1d252876fadb4e09aff9b1f8ace7cf56 2023-04-28 14:57:22,914 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/f1192348fc884b2283ab04f539b31400 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/f1192348fc884b2283ab04f539b31400 2023-04-28 14:57:22,915 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/956994df71a0433bb625328da3dd5142 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/956994df71a0433bb625328da3dd5142 2023-04-28 14:57:22,916 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/18c8c5124a7b436e8214367177e7761e to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/18c8c5124a7b436e8214367177e7761e 2023-04-28 14:57:22,917 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/088f581f7ce84f5a90640eac49332747 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/088f581f7ce84f5a90640eac49332747 2023-04-28 14:57:22,917 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/ba43580190f5438b98af185d3ef061c5 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/ba43580190f5438b98af185d3ef061c5 2023-04-28 14:57:22,919 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/737e68e8f0cf40cf8ba35195a357f97c to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/737e68e8f0cf40cf8ba35195a357f97c 2023-04-28 14:57:22,919 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/9002a2ab62644ca08949a617d26ea42b to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/9002a2ab62644ca08949a617d26ea42b 2023-04-28 14:57:22,920 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/ea4ae34ca6cc4fcf861d674f345e2685 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/ea4ae34ca6cc4fcf861d674f345e2685 2023-04-28 14:57:22,921 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/564671477c2842e0ac5ddfa9c9fdca0e to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/564671477c2842e0ac5ddfa9c9fdca0e 2023-04-28 14:57:22,922 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/09ce35b392494a8290ecad18442c98c0 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/09ce35b392494a8290ecad18442c98c0 2023-04-28 14:57:22,923 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/a415c8043f54483bba57b839146993bd to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/a415c8043f54483bba57b839146993bd 2023-04-28 14:57:22,924 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/af3ea4cb90a2466aa4d09f1ed5aba1b7 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/af3ea4cb90a2466aa4d09f1ed5aba1b7 2023-04-28 14:57:22,925 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/f5583a948f85434bbfee3da773638a00 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/f5583a948f85434bbfee3da773638a00 2023-04-28 14:57:22,927 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/5120d6cfc9024c939c163044613b68ec to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/5120d6cfc9024c939c163044613b68ec 2023-04-28 14:57:22,928 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/10a2d798647142d1958a56ca1bca2b56 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/10a2d798647142d1958a56ca1bca2b56 2023-04-28 14:57:22,930 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/bfce4cd51e414cc9a28a84eb2275dfc1 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/bfce4cd51e414cc9a28a84eb2275dfc1 2023-04-28 14:57:22,932 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/ed5231d71afe41b689a78acdbf835d7e to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/ed5231d71afe41b689a78acdbf835d7e 2023-04-28 14:57:22,933 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/9c5c8cee643c476ea289423ce15a06e6 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/9c5c8cee643c476ea289423ce15a06e6 2023-04-28 14:57:22,934 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/96b2335a1df747d58cf8e0a55fa377a0 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/96b2335a1df747d58cf8e0a55fa377a0 2023-04-28 14:57:22,935 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/d583e095bb10461a816d1b64aab00740 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/d583e095bb10461a816d1b64aab00740 2023-04-28 14:57:22,936 DEBUG [StoreCloser-TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/30e275e6aaed4428a5ff38abee7767ef to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/30e275e6aaed4428a5ff38abee7767ef 2023-04-28 14:57:22,940 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/recovered.edits/601.seqid, newMaxSeqId=601, maxSeqId=1 2023-04-28 14:57:22,942 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b. 2023-04-28 14:57:22,942 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1558): Region close journal for cce278287965607462f53e33830eba7b: 2023-04-28 14:57:22,943 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(147): Closed cce278287965607462f53e33830eba7b 2023-04-28 14:57:22,944 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=cce278287965607462f53e33830eba7b, regionState=CLOSED 2023-04-28 14:57:22,944 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693842944"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682693842944"}]},"ts":"1682693842944"} 2023-04-28 14:57:22,947 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-28 14:57:22,947 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure cce278287965607462f53e33830eba7b, server=jenkins-hbase17.apache.org,37181,1682693812482 in 1.7960 sec 2023-04-28 14:57:22,948 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-28 14:57:22,948 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=cce278287965607462f53e33830eba7b, UNASSIGN in 1.8010 sec 2023-04-28 14:57:22,949 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682693842949"}]},"ts":"1682693842949"} 2023-04-28 14:57:22,950 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-28 14:57:22,951 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-28 14:57:22,952 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.8150 sec 2023-04-28 14:57:23,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-28 14:57:23,253 INFO [Listener at localhost.localdomain/40193] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-28 14:57:23,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.HMaster$5(2228): Client=jenkins//136.243.18.41 delete TestAcidGuarantees 2023-04-28 14:57:23,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:57:23,266 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:57:23,268 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:57:23,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-28 14:57:23,273 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b 2023-04-28 14:57:23,276 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A, FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B, FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C, FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/recovered.edits] 2023-04-28 14:57:23,279 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/18f7133fa5b445fd8241cb7d5370b487 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/18f7133fa5b445fd8241cb7d5370b487 2023-04-28 14:57:23,280 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/65f0be0b151340a9a56f79a107f32ff2 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/65f0be0b151340a9a56f79a107f32ff2 2023-04-28 14:57:23,281 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/ce0f314a00984e43b5d87292ae0aca25 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/A/ce0f314a00984e43b5d87292ae0aca25 2023-04-28 14:57:23,284 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/39fc03dbfbc14a55af36c57a747e6195 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/39fc03dbfbc14a55af36c57a747e6195 2023-04-28 14:57:23,285 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/4f6636ca9ac84bf5ad90b20abf8364dd to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/4f6636ca9ac84bf5ad90b20abf8364dd 2023-04-28 14:57:23,287 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/ef2ec2d595694272843fe0a37ae780db to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/B/ef2ec2d595694272843fe0a37ae780db 2023-04-28 14:57:23,289 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/0beaef79f290476f86208b332c07c948 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/0beaef79f290476f86208b332c07c948 2023-04-28 14:57:23,291 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/4e21da5813514ab180044f23b9985f6d to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/4e21da5813514ab180044f23b9985f6d 2023-04-28 14:57:23,292 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/94cab253e34745aea2492787fb63e122 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/C/94cab253e34745aea2492787fb63e122 2023-04-28 14:57:23,295 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/recovered.edits/601.seqid to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b/recovered.edits/601.seqid 2023-04-28 14:57:23,296 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/cce278287965607462f53e33830eba7b 2023-04-28 14:57:23,296 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 14:57:23,298 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:57:23,324 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-28 14:57:23,366 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-28 14:57:23,368 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:57:23,368 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-28 14:57:23,368 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682693843368"}]},"ts":"9223372036854775807"} 2023-04-28 14:57:23,371 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-28 14:57:23,371 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => cce278287965607462f53e33830eba7b, NAME => 'TestAcidGuarantees,,1682693815816.cce278287965607462f53e33830eba7b.', STARTKEY => '', ENDKEY => ''}] 2023-04-28 14:57:23,371 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-28 14:57:23,371 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682693843371"}]},"ts":"9223372036854775807"} 2023-04-28 14:57:23,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-28 14:57:23,373 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-28 14:57:23,375 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:57:23,376 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 116 msec 2023-04-28 14:57:23,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-28 14:57:23,573 INFO [Listener at localhost.localdomain/40193] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-28 14:57:23,589 INFO [Listener at localhost.localdomain/40193] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=235 (was 207) Potentially hanging thread: (jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-676876121_15 at /127.0.0.1:42442 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1ad43520-shared-pool-26 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-676876121_15 at /127.0.0.1:42388 [Waiting for operation #6] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1ad43520-shared-pool-28 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-676876121_15 at /127.0.0.1:42510 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1ad43520-shared-pool-27 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase17:37181-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1ad43520-shared-pool-25 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=553 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=353 (was 261) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=5856 (was 5231) - AvailableMemoryMB LEAK? - 2023-04-28 14:57:23,599 INFO [Listener at localhost.localdomain/40193] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=235, OpenFileDescriptor=553, MaxFileDescriptor=60000, SystemLoadAverage=353, ProcessCount=168, AvailableMemoryMB=5853 2023-04-28 14:57:23,601 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-28 14:57:23,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.HMaster$4(2112): Client=jenkins//136.243.18.41 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-28 14:57:23,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-28 14:57:23,604 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-28 14:57:23,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(697): Client=jenkins//136.243.18.41 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-28 14:57:23,605 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-28 14:57:23,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-28 14:57:23,606 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:23,607 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b empty. 2023-04-28 14:57:23,607 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:23,607 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 14:57:23,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-28 14:57:23,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-28 14:57:24,017 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-28 14:57:24,019 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 3df9cf7b9785edd3a0db773c7f6b886b, NAME => 'TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp 2023-04-28 14:57:24,029 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 14:57:24,029 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 3df9cf7b9785edd3a0db773c7f6b886b, disabling compactions & flushes 2023-04-28 14:57:24,029 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:24,029 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:24,030 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. after waiting 0 ms 2023-04-28 14:57:24,030 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:24,030 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:24,030 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:24,032 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-28 14:57:24,033 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693844033"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682693844033"}]},"ts":"1682693844033"} 2023-04-28 14:57:24,034 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-28 14:57:24,035 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-28 14:57:24,035 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682693844035"}]},"ts":"1682693844035"} 2023-04-28 14:57:24,036 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-28 14:57:24,038 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3df9cf7b9785edd3a0db773c7f6b886b, ASSIGN}] 2023-04-28 14:57:24,040 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3df9cf7b9785edd3a0db773c7f6b886b, ASSIGN 2023-04-28 14:57:24,041 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=3df9cf7b9785edd3a0db773c7f6b886b, ASSIGN; state=OFFLINE, location=jenkins-hbase17.apache.org,37181,1682693812482; forceNewPlan=false, retain=false 2023-04-28 14:57:24,192 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=3df9cf7b9785edd3a0db773c7f6b886b, regionState=OPENING, regionLocation=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:24,192 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693844192"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682693844192"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682693844192"}]},"ts":"1682693844192"} 2023-04-28 14:57:24,194 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482}] 2023-04-28 14:57:24,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-28 14:57:24,351 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:24,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 3df9cf7b9785edd3a0db773c7f6b886b, NAME => 'TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.', STARTKEY => '', ENDKEY => ''} 2023-04-28 14:57:24,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:24,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 14:57:24,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7894): checking encryption for 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:24,352 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7897): checking classloading for 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:24,353 INFO [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:24,354 DEBUG [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A 2023-04-28 14:57:24,354 DEBUG [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A 2023-04-28 14:57:24,355 INFO [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 14:57:24,355 INFO [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3df9cf7b9785edd3a0db773c7f6b886b columnFamilyName A 2023-04-28 14:57:24,356 INFO [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] regionserver.HStore(310): Store=3df9cf7b9785edd3a0db773c7f6b886b/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:57:24,356 INFO [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:24,357 DEBUG [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B 2023-04-28 14:57:24,357 DEBUG [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B 2023-04-28 14:57:24,358 INFO [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 14:57:24,358 INFO [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3df9cf7b9785edd3a0db773c7f6b886b columnFamilyName B 2023-04-28 14:57:24,358 INFO [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] regionserver.HStore(310): Store=3df9cf7b9785edd3a0db773c7f6b886b/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:57:24,359 INFO [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:24,360 DEBUG [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C 2023-04-28 14:57:24,360 DEBUG [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C 2023-04-28 14:57:24,360 INFO [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 14:57:24,360 INFO [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3df9cf7b9785edd3a0db773c7f6b886b columnFamilyName C 2023-04-28 14:57:24,361 INFO [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] regionserver.HStore(310): Store=3df9cf7b9785edd3a0db773c7f6b886b/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:57:24,361 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:24,362 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:24,362 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:24,364 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 14:57:24,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1055): writing seq id for 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:24,367 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-28 14:57:24,367 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1072): Opened 3df9cf7b9785edd3a0db773c7f6b886b; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62265294, jitterRate=-0.07217481732368469}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 14:57:24,368 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(965): Region open journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:24,368 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b., pid=18, masterSystemTime=1682693844347 2023-04-28 14:57:24,369 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:24,370 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:24,370 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=3df9cf7b9785edd3a0db773c7f6b886b, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:24,370 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693844370"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1682693844370"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682693844370"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682693844370"}]},"ts":"1682693844370"} 2023-04-28 14:57:24,373 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-28 14:57:24,373 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 in 177 msec 2023-04-28 14:57:24,374 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-28 14:57:24,374 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3df9cf7b9785edd3a0db773c7f6b886b, ASSIGN in 335 msec 2023-04-28 14:57:24,375 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-28 14:57:24,375 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682693844375"}]},"ts":"1682693844375"} 2023-04-28 14:57:24,376 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-28 14:57:24,378 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-28 14:57:24,379 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 777 msec 2023-04-28 14:57:24,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-28 14:57:24,710 INFO [Listener at localhost.localdomain/40193] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-28 14:57:24,711 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x556231fa to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:57:24,716 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@671fd08e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:57:24,718 DEBUG [Listener at localhost.localdomain/40193] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:57:24,723 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:48920, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:57:24,725 DEBUG [Listener at localhost.localdomain/40193] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-28 14:57:24,729 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:56384, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-28 14:57:24,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-28 14:57:24,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.HMaster$12(2454): Client=jenkins//136.243.18.41 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-28 14:57:24,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-28 14:57:25,164 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-28 14:57:25,165 DEBUG [PEWorker-1] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-28 14:57:25,165 INFO [PEWorker-1] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-28 14:57:25,170 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-28 14:57:25,185 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3df9cf7b9785edd3a0db773c7f6b886b, REOPEN/MOVE}] 2023-04-28 14:57:25,186 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3df9cf7b9785edd3a0db773c7f6b886b, REOPEN/MOVE 2023-04-28 14:57:25,187 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=3df9cf7b9785edd3a0db773c7f6b886b, regionState=CLOSING, regionLocation=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:25,187 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693845187"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682693845187"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682693845187"}]},"ts":"1682693845187"} 2023-04-28 14:57:25,189 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482}] 2023-04-28 14:57:25,341 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(109): Close 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:25,342 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1604): Closing 3df9cf7b9785edd3a0db773c7f6b886b, disabling compactions & flushes 2023-04-28 14:57:25,342 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:25,342 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:25,342 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. after waiting 0 ms 2023-04-28 14:57:25,342 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:25,353 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-28 14:57:25,354 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:25,354 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1558): Region close journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:25,354 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 3df9cf7b9785edd3a0db773c7f6b886b to self. 2023-04-28 14:57:25,356 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(147): Closed 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:25,356 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=3df9cf7b9785edd3a0db773c7f6b886b, regionState=CLOSED 2023-04-28 14:57:25,356 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693845356"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682693845356"}]},"ts":"1682693845356"} 2023-04-28 14:57:25,359 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-28 14:57:25,360 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 in 169 msec 2023-04-28 14:57:25,360 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=3df9cf7b9785edd3a0db773c7f6b886b, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase17.apache.org,37181,1682693812482; forceNewPlan=false, retain=true 2023-04-28 14:57:25,511 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=3df9cf7b9785edd3a0db773c7f6b886b, regionState=OPENING, regionLocation=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:25,511 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693845510"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682693845510"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682693845510"}]},"ts":"1682693845510"} 2023-04-28 14:57:25,513 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482}] 2023-04-28 14:57:25,673 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:25,674 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 3df9cf7b9785edd3a0db773c7f6b886b, NAME => 'TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.', STARTKEY => '', ENDKEY => ''} 2023-04-28 14:57:25,674 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:25,674 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 14:57:25,674 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7894): checking encryption for 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:25,674 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7897): checking classloading for 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:25,682 INFO [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:25,684 DEBUG [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A 2023-04-28 14:57:25,684 DEBUG [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A 2023-04-28 14:57:25,684 INFO [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 14:57:25,689 INFO [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3df9cf7b9785edd3a0db773c7f6b886b columnFamilyName A 2023-04-28 14:57:25,692 INFO [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] regionserver.HStore(310): Store=3df9cf7b9785edd3a0db773c7f6b886b/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:57:25,693 INFO [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:25,694 DEBUG [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B 2023-04-28 14:57:25,694 DEBUG [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B 2023-04-28 14:57:25,694 INFO [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 14:57:25,695 INFO [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3df9cf7b9785edd3a0db773c7f6b886b columnFamilyName B 2023-04-28 14:57:25,697 INFO [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] regionserver.HStore(310): Store=3df9cf7b9785edd3a0db773c7f6b886b/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:57:25,698 INFO [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:25,700 DEBUG [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C 2023-04-28 14:57:25,700 DEBUG [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C 2023-04-28 14:57:25,700 INFO [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 14:57:25,701 INFO [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3df9cf7b9785edd3a0db773c7f6b886b columnFamilyName C 2023-04-28 14:57:25,702 INFO [StoreOpener-3df9cf7b9785edd3a0db773c7f6b886b-1] regionserver.HStore(310): Store=3df9cf7b9785edd3a0db773c7f6b886b/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:57:25,702 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:25,704 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:25,705 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:25,707 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 14:57:25,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1055): writing seq id for 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:25,709 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1072): Opened 3df9cf7b9785edd3a0db773c7f6b886b; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61281604, jitterRate=-0.0868329405784607}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 14:57:25,709 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(965): Region open journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:25,710 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b., pid=23, masterSystemTime=1682693845664 2023-04-28 14:57:25,711 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:25,712 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:25,712 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=3df9cf7b9785edd3a0db773c7f6b886b, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:25,712 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693845712"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1682693845712"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682693845712"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682693845712"}]},"ts":"1682693845712"} 2023-04-28 14:57:25,716 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-28 14:57:25,716 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 in 201 msec 2023-04-28 14:57:25,717 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-28 14:57:25,717 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3df9cf7b9785edd3a0db773c7f6b886b, REOPEN/MOVE in 531 msec 2023-04-28 14:57:25,720 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-28 14:57:25,720 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 549 msec 2023-04-28 14:57:25,722 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 980 msec 2023-04-28 14:57:25,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-28 14:57:25,731 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x389356cd to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:57:25,737 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@df763e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:57:25,738 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x595355a0 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:57:25,744 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7acc1387, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:57:25,745 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x38da9522 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:57:25,749 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7be739cc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:57:25,749 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x0a84dba6 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:57:25,758 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@486fba63, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:57:25,758 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x7c0b7748 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:57:25,765 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50feca84, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:57:25,765 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x0bd79dff to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:57:25,773 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d5bdceb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:57:25,773 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x52902392 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:57:25,778 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@651f745a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:57:25,778 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x4e1da5ba to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:57:25,784 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d429358, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:57:25,784 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x17a1bd04 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:57:25,792 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3432fa3f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:57:25,804 DEBUG [hconnection-0x2f2c17da-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:57:25,810 DEBUG [hconnection-0xd4c4d90-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:57:25,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:25,812 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:48932, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:57:25,817 DEBUG [hconnection-0x4b6272e1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:57:25,818 DEBUG [hconnection-0x66efef8b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:57:25,819 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:48950, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:57:25,825 DEBUG [hconnection-0x1f0c51a1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:57:25,833 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:48944, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:57:25,833 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:48976, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:57:25,834 DEBUG [hconnection-0x6bfce676-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:57:25,835 DEBUG [hconnection-0x56557344-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:57:25,836 DEBUG [hconnection-0x70ae915f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:57:25,837 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:48992, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:57:25,849 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:48948, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:57:25,850 DEBUG [hconnection-0x30f60230-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:57:25,856 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:48996, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:57:25,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:25,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:25,872 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:48986, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:57:25,872 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:25,872 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:25,872 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:25,872 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:25,872 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:48998, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:57:25,873 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:25,873 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:25,875 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:25,876 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:25,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:25,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:25,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:25,877 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:25,877 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:25,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:25,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:25,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:25,878 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:25,895 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:25,902 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:25,903 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:25,903 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:25,903 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:25,903 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:25,903 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:25,903 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:25,903 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:25,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:25,905 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3df9cf7b9785edd3a0db773c7f6b886b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 14:57:25,905 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:25,905 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:25,905 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:25,905 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:25,906 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:25,906 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:25,906 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:25,907 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:25,907 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:25,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:25,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:25,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:25,907 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:25,907 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:25,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:25,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:25,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:25,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:25,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:25,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:25,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48f8939e[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:25,908 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:25,908 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:25,909 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:25,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:25,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:25,909 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:25,909 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:25,909 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:25,909 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:25,909 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:25,914 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:25,914 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:25,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:25,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:25,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:25,914 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:25,915 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:25,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:25,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:25,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:25,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:25,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:25,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:25,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:25,918 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=A 2023-04-28 14:57:25,918 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:25,918 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=B 2023-04-28 14:57:25,918 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:25,918 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=C 2023-04-28 14:57:25,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:25,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:25,919 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:25,919 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:25,919 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:25,919 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:25,919 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:25,918 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:25,921 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:25,921 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:25,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:25,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:25,921 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:25,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:25,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:25,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:25,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:25,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:25,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:25,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:25,923 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:25,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:25,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:25,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:25,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:25,925 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:25,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:25,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:25,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:25,928 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:25,928 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:25,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:25,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:25,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:25,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:25,929 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:25,929 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:25,929 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:25,929 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:25,929 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:25,929 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:25,929 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:25,929 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:25,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:25,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:25,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:25,930 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:25,930 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:25,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:25,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48996 deadline: 1682693905946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:25,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682693845938, max timestamp=1682693845949Num uniques 859; 2023-04-28 14:57:25,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-28 14:57:25,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682693845938, max timestamp=1682693845949Num uniques 859; 2023-04-28 14:57:25,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-28 14:57:25,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693845944, max timestamp=1682693845954Num uniques 859; 2023-04-28 14:57:25,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:57:25,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:25,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48932 deadline: 1682693905989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:25,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:25,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48998 deadline: 1682693905990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:25,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:25,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48948 deadline: 1682693905992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:25,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:25,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693905992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,008 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428f8318b5a60314c5fb54d5b5cc928fa70 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428f8318b5a60314c5fb54d5b5cc928fa70 2023-04-28 14:57:26,030 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:26,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:26,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:26,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:26,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48996 deadline: 1682693906051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,071 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/252e41c384574c9ab43ccd78f8197e8a 2023-04-28 14:57:26,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:26,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:26,084 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:26,084 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:26,084 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:26,084 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:26,085 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:26,085 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:26,086 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:26,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:26,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:26,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:26,086 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:26,087 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,087 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:26,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:26,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:26,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:26,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:26,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48998 deadline: 1682693906097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,098 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:26,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693906097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,098 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:26,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48948 deadline: 1682693906097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:26,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48932 deadline: 1682693906102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,105 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:26,106 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:26,106 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:26,106 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:26,106 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:26,142 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:26,142 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:26,142 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:26,142 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:26,142 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:26,142 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:26,142 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:26,142 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:26,142 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,159 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:26,159 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:26,160 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,161 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:26,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:26,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:26,161 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:26,161 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:26,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:26,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:26,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:26,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:26,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:26,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@617f2440[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:26,163 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:26,163 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:26,164 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:26,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:26,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:26,164 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:26,164 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:26,164 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:26,164 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,164 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:26,165 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,165 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:26,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,165 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:26,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:26,165 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:26,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:26,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:26,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:26,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:26,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:26,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:26,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:26,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:26,168 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:26,168 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:26,168 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:26,168 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:26,168 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:26,169 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:26,169 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:26,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:26,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:26,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:26,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:26,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:26,169 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:26,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:26,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:26,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:26,170 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:26,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:26,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:26,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:26,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,172 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:26,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:26,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,174 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:26,174 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:26,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:26,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:26,174 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:26,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:26,174 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:26,174 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:26,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:26,174 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:26,174 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:26,174 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:26,174 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,175 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:26,175 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:26,175 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:26,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:26,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:26,210 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/2d4e4ea21f8547709e172516470d51e1 2023-04-28 14:57:26,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:26,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48996 deadline: 1682693906254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,275 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:26,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:26,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:26,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:26,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:26,294 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:26,294 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:26,294 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:26,294 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:26,294 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:26,295 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:26,295 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:26,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:26,295 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:26,296 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:26,296 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:26,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:26,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:26,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:26,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:26,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:26,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48998 deadline: 1682693906299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,301 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:26,301 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:26,301 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:26,301 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:26,301 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:26,301 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:26,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48948 deadline: 1682693906300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:26,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693906300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,311 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:26,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48932 deadline: 1682693906308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,312 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/96d4bea47b964c9985d562087b9e3fde 2023-04-28 14:57:26,318 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:26,318 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:26,318 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/252e41c384574c9ab43ccd78f8197e8a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/252e41c384574c9ab43ccd78f8197e8a 2023-04-28 14:57:26,322 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:26,322 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:26,323 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:26,323 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:26,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:26,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:26,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:26,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:26,324 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/252e41c384574c9ab43ccd78f8197e8a, entries=150, sequenceid=16, filesize=25.1 K 2023-04-28 14:57:26,324 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,324 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:26,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:26,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:26,325 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:26,325 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:26,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,326 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/2d4e4ea21f8547709e172516470d51e1 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/2d4e4ea21f8547709e172516470d51e1 2023-04-28 14:57:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:26,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:26,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b616a5e[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:26,327 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:26,328 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:26,328 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:26,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:26,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:26,328 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:26,328 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:26,328 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:26,329 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,329 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:26,329 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,329 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:26,330 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:26,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:26,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:26,330 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:26,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:26,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:26,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:26,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:26,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:26,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:26,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:26,332 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:26,332 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:26,332 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:26,332 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:26,332 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:26,333 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:26,334 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:26,334 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:26,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:26,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:26,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:26,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:26,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:26,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:26,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:26,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:26,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:26,338 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:26,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:26,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:26,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,339 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/2d4e4ea21f8547709e172516470d51e1, entries=150, sequenceid=16, filesize=11.6 K 2023-04-28 14:57:26,340 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:26,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:26,340 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/96d4bea47b964c9985d562087b9e3fde as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/96d4bea47b964c9985d562087b9e3fde 2023-04-28 14:57:26,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,342 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:26,342 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:26,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:26,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:26,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:26,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:26,343 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,344 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:26,344 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:26,344 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,344 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:26,344 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:26,344 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:26,344 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:26,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:26,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:26,344 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:26,344 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:26,356 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/96d4bea47b964c9985d562087b9e3fde, entries=150, sequenceid=16, filesize=11.6 K 2023-04-28 14:57:26,357 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=131.94 KB/135110 for 3df9cf7b9785edd3a0db773c7f6b886b in 451ms, sequenceid=16, compaction requested=false 2023-04-28 14:57:26,357 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-28 14:57:26,358 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:26,444 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:26,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:26,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:26,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:26,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:26,467 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:26,467 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:26,467 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:26,467 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:26,469 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:26,469 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:26,469 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:26,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:26,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:26,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:26,470 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:26,471 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,471 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:26,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:26,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:26,471 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:26,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:26,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:26,480 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:26,480 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:26,480 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:26,506 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:26,506 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:26,506 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:26,506 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:26,508 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3df9cf7b9785edd3a0db773c7f6b886b 3/3 column families, dataSize=131.94 KB heapSize=365.84 KB 2023-04-28 14:57:26,508 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=A 2023-04-28 14:57:26,508 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:26,508 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=B 2023-04-28 14:57:26,508 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:26,508 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=C 2023-04-28 14:57:26,508 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:26,568 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:26,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:26,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:26,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48996 deadline: 1682693906595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:26,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48998 deadline: 1682693906602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,607 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:26,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693906605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,607 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:26,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48948 deadline: 1682693906605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:26,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48932 deadline: 1682693906614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:26,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48996 deadline: 1682693906697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:26,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48996 deadline: 1682693906901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:26,972 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304283fab8fe867aa40a4aa5d1a39fad6eb4b to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304283fab8fe867aa40a4aa5d1a39fad6eb4b 2023-04-28 14:57:26,983 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/eadb01fa3f6e46e4913b35c9fa56d2d2 2023-04-28 14:57:27,106 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:27,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48998 deadline: 1682693907105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:27,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:27,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693907110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:27,114 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:27,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48948 deadline: 1682693907112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:27,118 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:27,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48932 deadline: 1682693907118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:27,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:27,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48996 deadline: 1682693907209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:27,407 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/7e09a7156a494124bc73b5cefa1fd02b 2023-04-28 14:57:27,717 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:27,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48996 deadline: 1682693907716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:27,824 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/6ec64e2eb42047288c623f64ee1cebc1 2023-04-28 14:57:27,828 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/eadb01fa3f6e46e4913b35c9fa56d2d2 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/eadb01fa3f6e46e4913b35c9fa56d2d2 2023-04-28 14:57:27,833 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/eadb01fa3f6e46e4913b35c9fa56d2d2, entries=150, sequenceid=50, filesize=25.1 K 2023-04-28 14:57:27,834 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/7e09a7156a494124bc73b5cefa1fd02b as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/7e09a7156a494124bc73b5cefa1fd02b 2023-04-28 14:57:27,842 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/7e09a7156a494124bc73b5cefa1fd02b, entries=150, sequenceid=50, filesize=11.6 K 2023-04-28 14:57:27,843 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/6ec64e2eb42047288c623f64ee1cebc1 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/6ec64e2eb42047288c623f64ee1cebc1 2023-04-28 14:57:27,847 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/6ec64e2eb42047288c623f64ee1cebc1, entries=150, sequenceid=50, filesize=11.6 K 2023-04-28 14:57:27,848 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.94 KB/135110, heapSize ~365.80 KB/374576, currentSize=53.67 KB/54960 for 3df9cf7b9785edd3a0db773c7f6b886b in 1340ms, sequenceid=50, compaction requested=false 2023-04-28 14:57:27,849 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:27,849 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:27,849 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:27,849 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:27,849 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:27,850 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:27,850 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:27,851 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:27,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:27,851 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:27,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:27,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:27,851 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:27,851 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:27,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:27,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:27,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:27,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:27,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:27,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:27,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fe8da2b[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:27,853 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:27,853 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:27,858 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:27,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:27,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:27,859 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:27,859 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:27,860 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:27,860 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:27,860 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:27,871 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:27,871 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:27,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:27,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:27,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:27,871 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:27,871 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:27,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:27,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:27,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:27,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:27,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:27,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:27,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:27,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:27,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:27,874 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:27,875 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:27,875 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:27,875 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:27,875 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:27,877 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:27,877 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:27,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:27,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:27,878 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:27,877 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:27,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:27,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:27,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:27,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:27,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:27,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:27,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:27,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:27,880 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:27,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:27,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:27,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:27,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:27,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:27,883 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:27,883 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:27,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:27,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:27,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:27,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:27,885 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:27,885 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:27,885 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:27,885 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:27,885 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:27,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:27,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:27,885 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:27,885 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:27,885 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:27,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:27,885 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:27,885 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:27,985 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:27,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:27,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:27,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:27,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:27,990 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:27,990 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:27,990 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:27,990 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:27,990 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:27,990 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:27,991 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:27,991 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:27,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:27,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:27,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:27,992 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:27,992 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:27,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:27,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:27,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:27,993 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:27,998 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:27,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:27,998 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:27,998 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:27,998 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:28,006 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:28,006 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:28,018 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:28,018 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:28,018 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3df9cf7b9785edd3a0db773c7f6b886b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-28 14:57:28,018 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=A 2023-04-28 14:57:28,018 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:28,019 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=B 2023-04-28 14:57:28,019 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:28,019 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=C 2023-04-28 14:57:28,019 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:28,070 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304289d501027105144508129b8f78a0da224 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304289d501027105144508129b8f78a0da224 2023-04-28 14:57:28,091 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=61, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/5d573dc91ee74902b64ed84e4225ad37 2023-04-28 14:57:28,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:28,117 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:28,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9554, totalHeapSize=21976, min timestamp=1682693848156, max timestamp=1682693848188Num uniques 859; 2023-04-28 14:57:28,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=39324, new segment data size=9554, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:57:28,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9558, totalHeapSize=21976, min timestamp=1682693848156, max timestamp=1682693848188Num uniques 859; 2023-04-28 14:57:28,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=39328, new segment data size=9558, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:57:28,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682693848156, max timestamp=1682693848180Num uniques 859; 2023-04-28 14:57:28,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:57:28,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:28,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693908210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:28,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48998 deadline: 1682693908210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:28,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48948 deadline: 1682693908212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:28,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48932 deadline: 1682693908212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:28,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693908314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:28,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48998 deadline: 1682693908314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:28,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48948 deadline: 1682693908314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:28,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48932 deadline: 1682693908315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,486 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-28 14:57:28,505 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/1f6cf567d47546a7a39586cdf316e07a 2023-04-28 14:57:28,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:28,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48932 deadline: 1682693908520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:28,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693908520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:28,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48998 deadline: 1682693908521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:28,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48948 deadline: 1682693908522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,554 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/f371330e759f4a3baa73ee8500a30e9a 2023-04-28 14:57:28,562 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/5d573dc91ee74902b64ed84e4225ad37 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/5d573dc91ee74902b64ed84e4225ad37 2023-04-28 14:57:28,567 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/5d573dc91ee74902b64ed84e4225ad37, entries=150, sequenceid=61, filesize=25.1 K 2023-04-28 14:57:28,568 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/1f6cf567d47546a7a39586cdf316e07a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/1f6cf567d47546a7a39586cdf316e07a 2023-04-28 14:57:28,574 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/1f6cf567d47546a7a39586cdf316e07a, entries=150, sequenceid=61, filesize=11.6 K 2023-04-28 14:57:28,578 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/f371330e759f4a3baa73ee8500a30e9a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/f371330e759f4a3baa73ee8500a30e9a 2023-04-28 14:57:28,586 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/f371330e759f4a3baa73ee8500a30e9a, entries=150, sequenceid=61, filesize=11.6 K 2023-04-28 14:57:28,587 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 3df9cf7b9785edd3a0db773c7f6b886b in 569ms, sequenceid=61, compaction requested=true 2023-04-28 14:57:28,587 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:28,587 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:28,587 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:28,587 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:28,587 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,587 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:28,587 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:28,588 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,588 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:28,589 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:28,589 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:28,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:28,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4300c42b[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:28,591 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:28,591 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:28,591 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:28,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:28,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:28,592 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:28,592 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:28,592 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:28,592 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,592 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:28,593 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,593 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:28,593 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:28,593 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:28,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:28,595 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:28,595 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:28,595 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:28,595 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:28,595 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:28,596 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:28,597 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:28,597 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:28,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:28,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:28,598 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:28,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:28,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,602 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:28,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:28,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,604 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:28,604 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:28,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:28,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:28,605 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,605 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:28,605 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:28,605 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,605 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:28,605 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:28,605 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:28,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:28,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:28,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:28,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:28,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:28,605 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:28,605 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:28,605 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:28,706 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:28,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:28,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:28,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:28,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:28,714 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:28,714 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:28,714 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:28,714 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:28,714 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:28,715 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:28,715 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:28,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:28,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:28,716 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:28,717 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,717 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:28,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:28,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:28,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:28,718 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:28,718 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:28,718 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:28,718 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:28,718 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:28,718 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:28,719 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:28,719 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:28,719 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:28,719 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:28,719 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3df9cf7b9785edd3a0db773c7f6b886b 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-28 14:57:28,719 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=A 2023-04-28 14:57:28,719 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:28,719 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=B 2023-04-28 14:57:28,719 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:28,719 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=C 2023-04-28 14:57:28,720 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:28,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:28,727 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:28,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:28,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48996 deadline: 1682693908755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:28,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693908828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:28,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48932 deadline: 1682693908828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:28,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48948 deadline: 1682693908833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:28,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48998 deadline: 1682693908833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:28,863 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:28,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48996 deadline: 1682693908861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:29,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:29,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48996 deadline: 1682693909066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:29,175 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042874669d17751743738e75729c2ad5824b to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042874669d17751743738e75729c2ad5824b 2023-04-28 14:57:29,335 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:29,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693909331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:29,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:29,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48932 deadline: 1682693909338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:29,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:29,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48948 deadline: 1682693909341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:29,349 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:29,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48998 deadline: 1682693909346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:29,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:29,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48996 deadline: 1682693909371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:29,582 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=97, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/dbed45591ee34d3a8b79f88e1269e12c 2023-04-28 14:57:29,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:29,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48996 deadline: 1682693909875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:29,998 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/4e50eb4e744644678482602992129273 2023-04-28 14:57:30,012 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/d830cceff4cb46358098ae7898363319 2023-04-28 14:57:30,019 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/dbed45591ee34d3a8b79f88e1269e12c as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/dbed45591ee34d3a8b79f88e1269e12c 2023-04-28 14:57:30,024 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/dbed45591ee34d3a8b79f88e1269e12c, entries=150, sequenceid=97, filesize=25.1 K 2023-04-28 14:57:30,025 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/4e50eb4e744644678482602992129273 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/4e50eb4e744644678482602992129273 2023-04-28 14:57:30,034 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/4e50eb4e744644678482602992129273, entries=150, sequenceid=97, filesize=11.6 K 2023-04-28 14:57:30,035 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/d830cceff4cb46358098ae7898363319 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/d830cceff4cb46358098ae7898363319 2023-04-28 14:57:30,043 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/d830cceff4cb46358098ae7898363319, entries=150, sequenceid=97, filesize=11.6 K 2023-04-28 14:57:30,044 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 3df9cf7b9785edd3a0db773c7f6b886b in 1325ms, sequenceid=97, compaction requested=true 2023-04-28 14:57:30,044 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:30,044 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:30,044 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:30,044 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:30,044 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,045 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:30,045 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:30,046 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,046 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:30,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:30,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:30,047 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:30,047 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:30,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:30,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:30,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:30,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:30,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:30,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bc5b52f[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:30,048 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:30,048 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:30,049 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:30,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:30,049 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:30,049 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:30,049 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:30,049 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,049 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:30,050 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:30,050 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:30,050 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:30,050 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:30,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:30,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:30,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:30,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:30,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:30,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:30,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:30,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:30,054 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:30,054 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:30,054 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:30,054 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:30,055 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:30,056 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:30,056 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:30,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:30,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:30,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:30,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:30,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:30,056 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:30,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:30,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:30,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:30,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:30,057 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:30,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:30,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:30,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,059 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,062 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:30,062 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:30,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:30,063 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:30,063 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:30,063 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:30,063 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:30,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:30,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:30,063 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,063 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:30,063 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:30,063 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:30,063 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:30,063 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:30,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:30,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:30,163 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:30,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:30,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:30,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:30,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:30,172 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:30,172 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:30,172 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:30,172 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:30,173 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:30,173 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:30,174 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:30,174 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:30,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:30,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:30,175 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,175 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:30,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:30,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:30,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:30,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:30,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:30,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:30,176 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:30,176 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:30,176 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:30,177 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:30,177 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:30,177 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:30,177 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:30,177 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3df9cf7b9785edd3a0db773c7f6b886b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 14:57:30,177 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=A 2023-04-28 14:57:30,177 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:30,177 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=B 2023-04-28 14:57:30,177 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:30,177 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=C 2023-04-28 14:57:30,178 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:30,244 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428652df8b81c9a4e9bb77c138fd431ca75 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428652df8b81c9a4e9bb77c138fd431ca75 2023-04-28 14:57:30,270 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=107, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/216f73997392447ea0d97ec77c777c63 2023-04-28 14:57:30,306 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/164049f739a0405bbde34cf81276a5f7 2023-04-28 14:57:30,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:30,344 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:30,373 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:30,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693910370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693850362, max timestamp=1682693850371Num uniques 859; 2023-04-28 14:57:30,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693850362, max timestamp=1682693850371Num uniques 859; 2023-04-28 14:57:30,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:30,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:57:30,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48948 deadline: 1682693910371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:57:30,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693850362, max timestamp=1682693850371Num uniques 859; 2023-04-28 14:57:30,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:57:30,414 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:30,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48998 deadline: 1682693910411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,414 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:30,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48932 deadline: 1682693910412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:30,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693910474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:30,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48948 deadline: 1682693910475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:30,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48998 deadline: 1682693910515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:30,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48932 deadline: 1682693910515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,678 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:30,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693910677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,678 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:30,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48948 deadline: 1682693910678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:30,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48998 deadline: 1682693910717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:30,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48932 deadline: 1682693910717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,732 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/bb931a9b11de4518b41a948f65f8fb67 2023-04-28 14:57:30,736 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/216f73997392447ea0d97ec77c777c63 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/216f73997392447ea0d97ec77c777c63 2023-04-28 14:57:30,741 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/216f73997392447ea0d97ec77c777c63, entries=150, sequenceid=107, filesize=25.1 K 2023-04-28 14:57:30,742 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/164049f739a0405bbde34cf81276a5f7 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/164049f739a0405bbde34cf81276a5f7 2023-04-28 14:57:30,746 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/164049f739a0405bbde34cf81276a5f7, entries=150, sequenceid=107, filesize=11.6 K 2023-04-28 14:57:30,747 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/bb931a9b11de4518b41a948f65f8fb67 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/bb931a9b11de4518b41a948f65f8fb67 2023-04-28 14:57:30,751 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/bb931a9b11de4518b41a948f65f8fb67, entries=150, sequenceid=107, filesize=11.6 K 2023-04-28 14:57:30,752 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 3df9cf7b9785edd3a0db773c7f6b886b in 575ms, sequenceid=107, compaction requested=true 2023-04-28 14:57:30,752 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:30,752 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:30,752 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:30,752 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:30,752 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,752 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:30,752 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:30,753 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:30,753 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:30,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:30,754 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:30,754 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:30,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:30,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:30,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:30,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:30,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:30,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b860793[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:30,755 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:30,755 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:30,756 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:30,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:30,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:30,756 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:30,756 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:30,756 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:30,756 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,756 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:30,757 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,757 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:30,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:30,757 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:30,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:30,757 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:30,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:30,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:30,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:30,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:30,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:30,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:30,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:30,759 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:30,759 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:30,760 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:30,760 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:30,760 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:30,760 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:30,760 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:30,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:30,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:30,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:30,761 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:30,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:30,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:30,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:30,761 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:30,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:30,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:30,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:30,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:30,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:30,762 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:30,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:30,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,764 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:30,764 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,764 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:30,764 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:30,764 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,764 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:30,764 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:30,764 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:30,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:30,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:30,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:30,764 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:30,764 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:30,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:30,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:30,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:30,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:30,765 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:30,765 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:30,865 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:30,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:30,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:30,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:30,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:30,868 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:30,868 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:30,868 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:30,868 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:30,869 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:30,869 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:30,869 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,869 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:30,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:30,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:30,869 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,870 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:30,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:30,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:30,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:30,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:30,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:30,874 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:30,874 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:30,874 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:30,874 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:30,874 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:30,878 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:30,878 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:30,878 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:30,878 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:30,878 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3df9cf7b9785edd3a0db773c7f6b886b 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-28 14:57:30,879 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=A 2023-04-28 14:57:30,879 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:30,879 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=B 2023-04-28 14:57:30,879 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:30,879 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=C 2023-04-28 14:57:30,879 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:30,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:30,889 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:30,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:30,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48996 deadline: 1682693910923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,983 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:30,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693910979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:30,983 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:30,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48948 deadline: 1682693910980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:31,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:31,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48998 deadline: 1682693911022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:31,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:31,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48932 deadline: 1682693911022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:31,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:31,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48996 deadline: 1682693911025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:31,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:31,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48996 deadline: 1682693911231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:31,295 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304280c04dfe803d74c92b4af6d8b28478b54 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304280c04dfe803d74c92b4af6d8b28478b54 2023-04-28 14:57:31,485 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:31,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48948 deadline: 1682693911485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:31,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:31,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693911485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:31,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:31,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48998 deadline: 1682693911527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:31,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:31,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48932 deadline: 1682693911528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:31,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:31,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48996 deadline: 1682693911541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:31,715 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=143, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/15740794a58043d5bec0f7c3e8912888 2023-04-28 14:57:32,052 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:32,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48996 deadline: 1682693912051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:32,134 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/5849a96ff25d4af2a730788ff14bc7f9 2023-04-28 14:57:32,495 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:32,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693912491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:32,496 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:32,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48948 deadline: 1682693912496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:32,536 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:32,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48932 deadline: 1682693912534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:32,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:32,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48998 deadline: 1682693912539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:32,566 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/805edbc8e6a140c0a55503b0ab087c36 2023-04-28 14:57:32,571 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/15740794a58043d5bec0f7c3e8912888 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/15740794a58043d5bec0f7c3e8912888 2023-04-28 14:57:32,575 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/15740794a58043d5bec0f7c3e8912888, entries=150, sequenceid=143, filesize=25.3 K 2023-04-28 14:57:32,576 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/5849a96ff25d4af2a730788ff14bc7f9 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/5849a96ff25d4af2a730788ff14bc7f9 2023-04-28 14:57:32,580 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/5849a96ff25d4af2a730788ff14bc7f9, entries=150, sequenceid=143, filesize=11.8 K 2023-04-28 14:57:32,581 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/805edbc8e6a140c0a55503b0ab087c36 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/805edbc8e6a140c0a55503b0ab087c36 2023-04-28 14:57:32,588 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/805edbc8e6a140c0a55503b0ab087c36, entries=150, sequenceid=143, filesize=11.8 K 2023-04-28 14:57:32,589 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 3df9cf7b9785edd3a0db773c7f6b886b in 1711ms, sequenceid=143, compaction requested=true 2023-04-28 14:57:32,589 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:32,589 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:32,589 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:32,589 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:32,589 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:32,589 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:32,589 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:32,592 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:32,592 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:32,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:32,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:32,593 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:32,593 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:32,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:32,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2236a94a[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:32,594 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:32,594 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:32,595 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:32,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:32,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:32,595 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:32,595 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:32,595 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:32,595 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:32,595 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:32,596 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:32,596 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:32,596 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:32,596 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:32,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:32,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:32,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:32,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:32,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:32,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:32,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:32,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:32,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:32,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:32,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:32,599 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:32,600 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:32,600 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:32,600 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:32,600 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:32,601 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:32,601 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:32,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:32,601 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:32,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:32,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:32,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:32,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:32,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:32,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:32,602 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:32,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:32,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:32,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:32,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:32,603 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:32,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:32,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:32,605 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:32,605 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:32,605 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:32,605 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:32,605 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:32,605 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:32,605 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:32,605 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:32,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:32,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:32,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:32,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:32,605 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:32,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:32,605 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:32,605 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:32,605 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:32,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:32,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:32,706 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:32,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:32,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:32,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:32,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:32,710 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:32,710 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:32,710 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:32,710 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:32,710 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:32,710 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:32,711 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:32,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:32,711 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:32,711 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:32,711 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:32,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:32,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:32,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:32,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:32,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:32,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:32,712 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:32,712 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:32,712 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:32,713 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:32,713 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:32,713 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:32,713 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:32,713 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3df9cf7b9785edd3a0db773c7f6b886b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 14:57:32,713 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=A 2023-04-28 14:57:32,713 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:32,713 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=B 2023-04-28 14:57:32,713 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:32,713 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=C 2023-04-28 14:57:32,714 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:32,736 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304288ecc9b777ff7481689450a37d45a940c to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304288ecc9b777ff7481689450a37d45a940c 2023-04-28 14:57:33,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:33,071 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:33,142 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=153, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/d861af5d1fd043489ea9ae8b74363f33 2023-04-28 14:57:33,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693853134, max timestamp=1682693853146Num uniques 859; 2023-04-28 14:57:33,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:57:33,150 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693853134, max timestamp=1682693853146Num uniques 859; 2023-04-28 14:57:33,150 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:57:33,151 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682693853134, max timestamp=1682693853146Num uniques 859; 2023-04-28 14:57:33,151 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-28 14:57:33,163 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/12b9e9b6629c41d59efa5169e72ede33 2023-04-28 14:57:33,205 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/763afba87f634ba6ace64249776bb3e6 2023-04-28 14:57:33,212 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/d861af5d1fd043489ea9ae8b74363f33 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/d861af5d1fd043489ea9ae8b74363f33 2023-04-28 14:57:33,218 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/d861af5d1fd043489ea9ae8b74363f33, entries=100, sequenceid=153, filesize=18.5 K 2023-04-28 14:57:33,220 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/12b9e9b6629c41d59efa5169e72ede33 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/12b9e9b6629c41d59efa5169e72ede33 2023-04-28 14:57:33,226 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/12b9e9b6629c41d59efa5169e72ede33, entries=100, sequenceid=153, filesize=9.4 K 2023-04-28 14:57:33,227 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/763afba87f634ba6ace64249776bb3e6 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/763afba87f634ba6ace64249776bb3e6 2023-04-28 14:57:33,253 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/763afba87f634ba6ace64249776bb3e6, entries=100, sequenceid=153, filesize=9.4 K 2023-04-28 14:57:33,254 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=133.78 KB/136995 for 3df9cf7b9785edd3a0db773c7f6b886b in 541ms, sequenceid=153, compaction requested=true 2023-04-28 14:57:33,254 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:33,254 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:33,254 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:33,254 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:33,254 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,254 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:33,254 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:33,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682693853219, max timestamp=1682693853253Num uniques 1018; 2023-04-28 14:57:33,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682693853219, max timestamp=1682693853253Num uniques 1018; 2023-04-28 14:57:33,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-28 14:57:33,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-28 14:57:33,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682693853219, max timestamp=1682693853253Num uniques 1009; 2023-04-28 14:57:33,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-28 14:57:33,258 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:33,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:33,259 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:33,259 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,259 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:33,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:33,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:33,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:33,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:33,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@675210bb[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:33,260 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:33,260 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,261 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,261 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:33,261 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:33,261 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:33,262 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,262 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:33,263 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:33,263 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:33,263 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:33,264 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:33,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:33,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:33,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:33,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:33,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:33,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:33,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:33,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:33,267 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:33,267 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:33,268 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:33,268 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:33,268 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:33,269 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,269 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:33,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:33,270 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:33,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:33,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:33,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:33,272 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:33,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:33,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:33,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,274 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:33,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:33,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,276 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:33,276 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:33,276 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,276 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:33,276 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:33,276 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:33,276 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:33,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:33,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:33,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:33,276 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,277 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,277 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,277 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:33,277 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:33,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:33,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:33,289 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3df9cf7b9785edd3a0db773c7f6b886b 3/3 column families, dataSize=53.67 KB heapSize=145.08 KB 2023-04-28 14:57:33,289 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=A 2023-04-28 14:57:33,289 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:33,289 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=B 2023-04-28 14:57:33,289 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:33,289 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=C 2023-04-28 14:57:33,289 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:33,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:33,329 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042855dee64cc3bb449384808086f42a7b24 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042855dee64cc3bb449384808086f42a7b24 2023-04-28 14:57:33,336 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=195, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/2f1068b82a2440a6918bb6ae7297b602 2023-04-28 14:57:33,377 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:33,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:33,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:33,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693853363, max timestamp=1682693853387Num uniques 859; 2023-04-28 14:57:33,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693853363, max timestamp=1682693853387Num uniques 859; 2023-04-28 14:57:33,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:57:33,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:57:33,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693853363, max timestamp=1682693853387Num uniques 859; 2023-04-28 14:57:33,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:57:33,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:33,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:33,398 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:33,398 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:33,398 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:33,398 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:33,399 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,399 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:33,399 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,399 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:33,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:33,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:33,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:33,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,400 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,400 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:33,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:33,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:33,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:33,401 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:33,401 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:33,401 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:33,401 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:33,401 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:33,402 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:33,402 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:33,402 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:33,402 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:33,402 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:33,402 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:33,402 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:33,402 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:33,402 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,402 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:33,402 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:33,403 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,403 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:33,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:33,404 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,404 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:33,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:33,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:33,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:33,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:33,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:33,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11c8bdb3[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:33,405 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:33,405 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,406 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,406 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:33,406 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:33,406 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:33,406 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,406 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:33,408 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:33,408 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,408 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:33,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:33,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:33,408 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:33,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:33,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:33,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:33,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:33,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:33,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:33,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:33,411 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:33,411 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:33,411 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:33,411 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:33,412 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:33,414 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,414 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:33,414 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:33,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:33,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:33,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:33,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:33,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,415 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:33,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:33,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:33,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:33,416 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:33,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,419 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:33,419 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:33,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:33,419 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,419 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:33,419 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:33,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:33,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:33,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:33,419 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:33,419 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,419 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,419 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,419 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:33,419 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,420 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:33,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:33,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:33,466 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:33,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693913464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,520 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:33,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:33,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:33,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:33,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:33,528 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:33,528 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:33,528 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:33,528 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:33,529 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,529 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:33,529 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:33,530 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:33,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:33,530 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,530 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:33,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:33,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:33,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:33,531 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:33,531 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:33,531 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:33,531 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:33,531 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:33,531 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:33,531 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:33,531 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:33,531 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:33,531 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:33,531 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:33,531 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:33,532 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:33,532 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:33,532 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,532 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:33,532 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:33,533 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:33,533 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:33,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:33,533 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,533 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:33,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:33,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:33,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:33,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:33,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ae98a86[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:33,534 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:33,534 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,536 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,536 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:33,536 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,536 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:33,536 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:33,536 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:33,537 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,537 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:33,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:33,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:33,537 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:33,537 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:33,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:33,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:33,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:33,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:33,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:33,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:33,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:33,539 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:33,539 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:33,539 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:33,539 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:33,539 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:33,540 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,540 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:33,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:33,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,540 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:33,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:33,541 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:33,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:33,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:33,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:33,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:33,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,542 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:33,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:33,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,544 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,544 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:33,544 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:33,544 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:33,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:33,544 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:33,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:33,544 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:33,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:33,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:33,544 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,544 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,544 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,544 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,544 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:33,544 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:33,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:33,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:33,569 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:33,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693913568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,644 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:33,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:33,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:33,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:33,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:33,648 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:33,648 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:33,648 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:33,648 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:33,648 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,648 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:33,649 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,649 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:33,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:33,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:33,649 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,649 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:33,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:33,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:33,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:33,650 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:33,650 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:33,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:33,650 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:33,650 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:33,650 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:33,650 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:33,650 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:33,650 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:33,650 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:33,650 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:33,650 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:33,650 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:33,650 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:33,650 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,651 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:33,651 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:33,651 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,652 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,652 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:33,652 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:33,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:33,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:33,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:33,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:33,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:33,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:33,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d020e68[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:33,653 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:33,653 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,654 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,654 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:33,654 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:33,654 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:33,654 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,654 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:33,655 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:33,655 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,655 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:33,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:33,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:33,655 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:33,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:33,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:33,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:33,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:33,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:33,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:33,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:33,657 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:33,657 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:33,657 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:33,657 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:33,657 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:33,658 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,658 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:33,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:33,658 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:33,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:33,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:33,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:33,658 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:33,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:33,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:33,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,659 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:33,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:33,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,661 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,661 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:33,661 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:33,661 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:33,661 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:33,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:33,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:33,661 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:33,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:33,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:33,661 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:33,661 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,661 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,661 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:33,661 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,661 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:33,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:33,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:33,752 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/454095b669934edeabfc182e83ee7601 2023-04-28 14:57:33,761 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:33,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:33,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:33,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:33,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:33,765 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:33,765 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:33,765 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:33,765 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:33,766 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,766 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:33,766 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,767 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,767 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:33,767 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:33,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:33,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:33,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:33,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:33,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:33,768 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:33,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:33,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693913771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,782 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:33,782 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:33,782 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:33,782 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:33,782 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:33,783 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:33,783 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:33,783 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:33,784 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:33,784 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:33,784 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:33,784 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:33,784 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:33,784 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,784 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:33,784 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:33,785 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,785 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:33,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:33,785 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,785 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:33,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:33,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:33,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:33,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:33,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:33,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a2202bb[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:33,787 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:33,787 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,789 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,790 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:33,790 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:33,790 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:33,790 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,790 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:33,791 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:33,791 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,791 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:33,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:33,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:33,791 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:33,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:33,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:33,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:33,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:33,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:33,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:33,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:33,793 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:33,793 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:33,793 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:33,793 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:33,793 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:33,799 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,799 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:33,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,799 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:33,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:33,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:33,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:33,800 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:33,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:33,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:33,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:33,801 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:33,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:33,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,941 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,941 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:33,941 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:33,941 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:33,941 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:33,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:33,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:33,941 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:33,941 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:33,941 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,942 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:33,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:33,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:33,942 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:33,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:33,942 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:33,942 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:33,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:33,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:34,042 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:34,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:34,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:34,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:34,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:34,055 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:34,055 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:34,055 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:34,055 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:34,055 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:34,055 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:34,057 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:34,057 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:34,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:34,057 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,058 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:34,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:34,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:34,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:34,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:34,058 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:34,059 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:34,059 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:34,059 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:34,059 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:34,059 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:34,059 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:34,059 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:34,059 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:34,059 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:34,059 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:34,059 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:34,059 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:34,059 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:34,059 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,059 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:34,059 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:34,060 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:34,060 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:34,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:34,060 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:34,060 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:34,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:34,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:34,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:34,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:34,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:34,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ed30f69[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:34,062 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:34,062 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:34,062 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:34,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:34,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:34,062 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:34,062 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,062 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:34,062 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:34,062 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:34,063 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:34,063 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,063 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:34,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:34,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:34,064 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:34,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:34,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:34,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:34,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:34,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:34,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:34,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:34,066 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:34,066 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:34,066 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:34,066 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:34,066 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:34,067 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:34,067 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:34,067 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:34,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:34,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:34,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:34,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:34,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:34,067 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:34,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:34,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:34,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:34,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:34,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:34,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:34,068 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:34,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:34,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,070 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:34,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:34,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:34,070 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,070 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:34,070 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:34,070 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:34,070 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:34,070 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:34,070 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,070 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:34,070 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:34,070 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:34,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:34,070 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:34,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:34,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:34,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:34,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:34,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:34,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693914078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,167 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/46f9fd9f2c634804864c7de08e078af6 2023-04-28 14:57:34,170 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:34,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:34,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:34,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:34,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:34,176 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:34,176 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:34,176 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:34,176 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:34,176 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:34,176 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:34,177 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/2f1068b82a2440a6918bb6ae7297b602 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/2f1068b82a2440a6918bb6ae7297b602 2023-04-28 14:57:34,177 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:34,177 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:34,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:34,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:34,178 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,178 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:34,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:34,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:34,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:34,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:34,183 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/2f1068b82a2440a6918bb6ae7297b602, entries=200, sequenceid=195, filesize=32.1 K 2023-04-28 14:57:34,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:34,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:34,184 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:34,184 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:34,184 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:34,185 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/454095b669934edeabfc182e83ee7601 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/454095b669934edeabfc182e83ee7601 2023-04-28 14:57:34,190 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/454095b669934edeabfc182e83ee7601, entries=150, sequenceid=195, filesize=11.8 K 2023-04-28 14:57:34,191 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/46f9fd9f2c634804864c7de08e078af6 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/46f9fd9f2c634804864c7de08e078af6 2023-04-28 14:57:34,193 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:34,193 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:34,193 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:34,193 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:34,193 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:34,193 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:34,193 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:34,193 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:34,193 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,193 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:34,193 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:34,194 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,194 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:34,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:34,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:34,195 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:34,195 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:34,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:34,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:34,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:34,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:34,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:34,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78281f8f[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:34,196 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:34,196 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:34,197 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:34,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:34,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:34,197 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:34,197 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:34,197 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:34,197 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,197 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:34,198 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:34,198 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:34,198 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:34,199 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:34,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:34,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:34,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:34,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:34,206 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/46f9fd9f2c634804864c7de08e078af6, entries=150, sequenceid=195, filesize=11.8 K 2023-04-28 14:57:34,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:34,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:34,207 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:34,207 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:34,207 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:34,207 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:34,207 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:34,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:34,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:34,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:34,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:34,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:34,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:34,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:34,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:34,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:34,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:34,209 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:34,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:34,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:34,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:34,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:34,210 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~145.03 KB/148512, currentSize=134.18 KB/137400 for 3df9cf7b9785edd3a0db773c7f6b886b in 922ms, sequenceid=195, compaction requested=true 2023-04-28 14:57:34,210 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:34,210 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,210 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:34,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,210 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-28 14:57:34,211 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-28 14:57:34,211 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:34,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:34,212 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:57:34,212 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,212 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:34,212 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:34,212 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:34,212 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:34,212 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,213 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:34,213 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:34,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:34,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:34,213 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:34,213 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:34,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:34,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:34,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:34,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:34,214 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:34,214 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:34,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:34,215 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 93144 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-28 14:57:34,215 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 3df9cf7b9785edd3a0db773c7f6b886b/B is initiating minor compaction (all files) 2023-04-28 14:57:34,215 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3df9cf7b9785edd3a0db773c7f6b886b/B in TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:34,215 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/2d4e4ea21f8547709e172516470d51e1, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/7e09a7156a494124bc73b5cefa1fd02b, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/1f6cf567d47546a7a39586cdf316e07a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/4e50eb4e744644678482602992129273, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/164049f739a0405bbde34cf81276a5f7, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/5849a96ff25d4af2a730788ff14bc7f9, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/12b9e9b6629c41d59efa5169e72ede33, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/454095b669934edeabfc182e83ee7601] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp, totalSize=91.0 K 2023-04-28 14:57:34,216 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 206410 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-28 14:57:34,216 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): 3df9cf7b9785edd3a0db773c7f6b886b/A is initiating minor compaction (all files) 2023-04-28 14:57:34,216 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3df9cf7b9785edd3a0db773c7f6b886b/A in TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:34,216 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/252e41c384574c9ab43ccd78f8197e8a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/eadb01fa3f6e46e4913b35c9fa56d2d2, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/5d573dc91ee74902b64ed84e4225ad37, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/dbed45591ee34d3a8b79f88e1269e12c, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/216f73997392447ea0d97ec77c777c63, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/15740794a58043d5bec0f7c3e8912888, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/d861af5d1fd043489ea9ae8b74363f33, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/2f1068b82a2440a6918bb6ae7297b602] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp, totalSize=201.6 K 2023-04-28 14:57:34,216 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 2d4e4ea21f8547709e172516470d51e1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682693845875 2023-04-28 14:57:34,217 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 252e41c384574c9ab43ccd78f8197e8a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682693845875 2023-04-28 14:57:34,217 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 7e09a7156a494124bc73b5cefa1fd02b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682693845987 2023-04-28 14:57:34,217 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting eadb01fa3f6e46e4913b35c9fa56d2d2, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682693845987 2023-04-28 14:57:34,218 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 1f6cf567d47546a7a39586cdf316e07a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1682693846578 2023-04-28 14:57:34,218 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 5d573dc91ee74902b64ed84e4225ad37, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1682693846578 2023-04-28 14:57:34,218 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 4e50eb4e744644678482602992129273, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1682693848208 2023-04-28 14:57:34,218 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting dbed45591ee34d3a8b79f88e1269e12c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1682693848208 2023-04-28 14:57:34,218 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 164049f739a0405bbde34cf81276a5f7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1682693848739 2023-04-28 14:57:34,219 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 5849a96ff25d4af2a730788ff14bc7f9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1682693850386 2023-04-28 14:57:34,219 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 216f73997392447ea0d97ec77c777c63, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1682693848739 2023-04-28 14:57:34,219 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 12b9e9b6629c41d59efa5169e72ede33, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1682693850920 2023-04-28 14:57:34,219 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 15740794a58043d5bec0f7c3e8912888, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1682693850386 2023-04-28 14:57:34,220 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 454095b669934edeabfc182e83ee7601, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682693853253 2023-04-28 14:57:34,220 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting d861af5d1fd043489ea9ae8b74363f33, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1682693850920 2023-04-28 14:57:34,220 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 2f1068b82a2440a6918bb6ae7297b602, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682693853253 2023-04-28 14:57:34,247 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 3df9cf7b9785edd3a0db773c7f6b886b#B#compaction#129 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:34,271 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3df9cf7b9785edd3a0db773c7f6b886b#A#compaction#130 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:34,281 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/0e47d7e42d1a4f0390880d8a800dcd16 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/0e47d7e42d1a4f0390880d8a800dcd16 2023-04-28 14:57:34,288 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 3df9cf7b9785edd3a0db773c7f6b886b/B of 3df9cf7b9785edd3a0db773c7f6b886b into 0e47d7e42d1a4f0390880d8a800dcd16(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:34,288 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:34,288 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b., storeName=3df9cf7b9785edd3a0db773c7f6b886b/B, priority=8, startTime=1682693854211; duration=0sec 2023-04-28 14:57:34,288 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:57:34,288 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-28 14:57:34,291 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 93144 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-28 14:57:34,291 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 3df9cf7b9785edd3a0db773c7f6b886b/C is initiating minor compaction (all files) 2023-04-28 14:57:34,291 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3df9cf7b9785edd3a0db773c7f6b886b/C in TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:34,291 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/96d4bea47b964c9985d562087b9e3fde, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/6ec64e2eb42047288c623f64ee1cebc1, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/f371330e759f4a3baa73ee8500a30e9a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/d830cceff4cb46358098ae7898363319, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/bb931a9b11de4518b41a948f65f8fb67, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/805edbc8e6a140c0a55503b0ab087c36, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/763afba87f634ba6ace64249776bb3e6, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/46f9fd9f2c634804864c7de08e078af6] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp, totalSize=91.0 K 2023-04-28 14:57:34,292 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 96d4bea47b964c9985d562087b9e3fde, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682693845875 2023-04-28 14:57:34,292 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 6ec64e2eb42047288c623f64ee1cebc1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682693845987 2023-04-28 14:57:34,292 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting f371330e759f4a3baa73ee8500a30e9a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1682693846578 2023-04-28 14:57:34,293 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting d830cceff4cb46358098ae7898363319, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1682693848208 2023-04-28 14:57:34,305 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting bb931a9b11de4518b41a948f65f8fb67, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1682693848739 2023-04-28 14:57:34,310 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 805edbc8e6a140c0a55503b0ab087c36, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1682693850386 2023-04-28 14:57:34,310 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 763afba87f634ba6ace64249776bb3e6, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1682693850920 2023-04-28 14:57:34,311 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 46f9fd9f2c634804864c7de08e078af6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682693853253 2023-04-28 14:57:34,314 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:34,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:34,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:34,325 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 3df9cf7b9785edd3a0db773c7f6b886b#C#compaction#131 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:34,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:34,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:34,332 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:34,333 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:34,333 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:34,333 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:34,333 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:34,333 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:34,334 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:34,334 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:34,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:34,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:34,334 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,334 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:34,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:34,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:34,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:34,335 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:34,342 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:34,342 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:34,342 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:34,342 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:34,342 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:34,346 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:34,347 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:34,347 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:34,347 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:34,347 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3df9cf7b9785edd3a0db773c7f6b886b 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-28 14:57:34,348 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=A 2023-04-28 14:57:34,348 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:34,348 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=B 2023-04-28 14:57:34,348 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:34,348 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=C 2023-04-28 14:57:34,348 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:34,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:34,509 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:34,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:34,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48948 deadline: 1682693914519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:34,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693914521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:34,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48998 deadline: 1682693914546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,554 DEBUG [Thread-808] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b., hostname=jenkins-hbase17.apache.org,37181,1682693812482, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 14:57:34,556 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:34,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48932 deadline: 1682693914553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,557 DEBUG [Thread-802] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b., hostname=jenkins-hbase17.apache.org,37181,1682693812482, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 14:57:34,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:34,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693914587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,624 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:34,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48948 deadline: 1682693914623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:34,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693914623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,739 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/25fc7e9715ad4320b85f7011c6e1b85d as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/25fc7e9715ad4320b85f7011c6e1b85d 2023-04-28 14:57:34,744 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 3df9cf7b9785edd3a0db773c7f6b886b/A of 3df9cf7b9785edd3a0db773c7f6b886b into 25fc7e9715ad4320b85f7011c6e1b85d(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:34,744 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:34,744 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b., storeName=3df9cf7b9785edd3a0db773c7f6b886b/A, priority=8, startTime=1682693854210; duration=0sec 2023-04-28 14:57:34,744 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:34,753 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/9a8f2e00703d4fd9b6a8c30d5ca4de8e as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/9a8f2e00703d4fd9b6a8c30d5ca4de8e 2023-04-28 14:57:34,758 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 3df9cf7b9785edd3a0db773c7f6b886b/C of 3df9cf7b9785edd3a0db773c7f6b886b into 9a8f2e00703d4fd9b6a8c30d5ca4de8e(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:34,758 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:34,758 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b., storeName=3df9cf7b9785edd3a0db773c7f6b886b/C, priority=8, startTime=1682693854212; duration=0sec 2023-04-28 14:57:34,758 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:34,774 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042802cb1a2c068c4b79b9a364d6a552afe3 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042802cb1a2c068c4b79b9a364d6a552afe3 2023-04-28 14:57:34,779 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=232, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/396612d922734853942f822e06397ed1 2023-04-28 14:57:34,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:34,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48948 deadline: 1682693914845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:34,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:34,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693914846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:35,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:35,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48948 deadline: 1682693915149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:35,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:35,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693915150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:35,188 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/a3095a4faa0c43d68cff36a4545ae1ad 2023-04-28 14:57:35,197 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/1c02482d3f744352b57022433de4edea 2023-04-28 14:57:35,201 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/396612d922734853942f822e06397ed1 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/396612d922734853942f822e06397ed1 2023-04-28 14:57:35,205 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/396612d922734853942f822e06397ed1, entries=150, sequenceid=232, filesize=25.3 K 2023-04-28 14:57:35,206 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/a3095a4faa0c43d68cff36a4545ae1ad as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/a3095a4faa0c43d68cff36a4545ae1ad 2023-04-28 14:57:35,209 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/a3095a4faa0c43d68cff36a4545ae1ad, entries=150, sequenceid=232, filesize=11.8 K 2023-04-28 14:57:35,210 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/1c02482d3f744352b57022433de4edea as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/1c02482d3f744352b57022433de4edea 2023-04-28 14:57:35,221 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/1c02482d3f744352b57022433de4edea, entries=150, sequenceid=232, filesize=11.8 K 2023-04-28 14:57:35,222 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 3df9cf7b9785edd3a0db773c7f6b886b in 875ms, sequenceid=232, compaction requested=false 2023-04-28 14:57:35,222 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:35,222 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:35,222 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:35,222 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:35,222 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:35,222 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:35,222 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:35,224 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:35,224 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:35,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:35,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:35,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:35,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:35,225 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:35,225 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:35,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:35,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:35,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:35,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:35,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:35,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75d43fac[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:35,227 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:35,227 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:35,227 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:35,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:35,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:35,228 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:35,228 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:35,228 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:35,228 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:35,228 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:35,229 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:35,229 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:35,229 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:35,229 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:35,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:35,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:35,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:35,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:35,231 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:35,231 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:35,231 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:35,232 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:35,232 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:35,233 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:35,233 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:35,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:35,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:35,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:35,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:35,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:35,233 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:35,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:35,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:35,233 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:35,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:35,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:35,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:35,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:35,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:35,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:35,235 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:35,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:35,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:35,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:35,236 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:35,236 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:35,236 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:35,236 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:35,236 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:35,236 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:35,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:35,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:35,236 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:35,236 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:35,237 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:35,237 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:35,238 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:35,238 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:35,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:35,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:35,337 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:35,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:35,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:35,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:35,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:35,341 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:35,341 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:35,341 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:35,341 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:35,341 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:35,341 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:35,343 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:35,343 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:35,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:35,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:35,343 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:35,343 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:35,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:35,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:35,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:35,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:35,344 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:35,344 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:35,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:35,344 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:35,345 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:35,345 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:35,345 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:35,345 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:35,345 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:35,345 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:35,345 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3df9cf7b9785edd3a0db773c7f6b886b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 14:57:35,345 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=A 2023-04-28 14:57:35,345 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:35,345 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=B 2023-04-28 14:57:35,345 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:35,345 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=C 2023-04-28 14:57:35,345 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:35,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:35,608 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:35,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682693855656, max timestamp=1682693855668Num uniques 859; 2023-04-28 14:57:35,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682693855656, max timestamp=1682693855668Num uniques 859; 2023-04-28 14:57:35,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-28 14:57:35,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-28 14:57:35,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682693855656, max timestamp=1682693855668Num uniques 859; 2023-04-28 14:57:35,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 14:57:35,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:35,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48948 deadline: 1682693915687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:35,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:35,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693915688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:35,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:35,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693915689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:35,762 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304283b8458e0ba374d4b960abee53dc7d08c to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304283b8458e0ba374d4b960abee53dc7d08c 2023-04-28 14:57:35,794 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:35,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48948 deadline: 1682693915793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:35,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:35,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693915793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:35,797 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:35,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693915796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:35,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:35,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48948 deadline: 1682693915996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,000 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:36,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693915997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,000 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:36,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693915998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,174 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=244, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/00d850e9e50c49cf8b0b7a1b524f8158 2023-04-28 14:57:36,184 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/f6c9de97a9504b49b28be51d5a67cd16 2023-04-28 14:57:36,192 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/c6d383d37a294f4886e897fc6e79b68b 2023-04-28 14:57:36,196 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/00d850e9e50c49cf8b0b7a1b524f8158 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/00d850e9e50c49cf8b0b7a1b524f8158 2023-04-28 14:57:36,199 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/00d850e9e50c49cf8b0b7a1b524f8158, entries=100, sequenceid=244, filesize=18.5 K 2023-04-28 14:57:36,200 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/f6c9de97a9504b49b28be51d5a67cd16 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/f6c9de97a9504b49b28be51d5a67cd16 2023-04-28 14:57:36,203 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/f6c9de97a9504b49b28be51d5a67cd16, entries=100, sequenceid=244, filesize=9.4 K 2023-04-28 14:57:36,204 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/c6d383d37a294f4886e897fc6e79b68b as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/c6d383d37a294f4886e897fc6e79b68b 2023-04-28 14:57:36,207 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/c6d383d37a294f4886e897fc6e79b68b, entries=100, sequenceid=244, filesize=9.4 K 2023-04-28 14:57:36,208 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=136.42 KB/139690 for 3df9cf7b9785edd3a0db773c7f6b886b in 863ms, sequenceid=244, compaction requested=true 2023-04-28 14:57:36,208 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:36,208 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:36,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:36,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:36,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:36,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:36,209 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,209 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:36,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:36,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:36,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:36,210 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,210 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:36,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:36,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:36,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f99dc77[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:36,211 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:36,211 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,211 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,212 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:36,212 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:36,212 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:36,212 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,212 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:36,214 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,214 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:36,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:36,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:36,214 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:36,214 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:36,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:36,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:36,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:36,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:36,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:36,217 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:36,217 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:36,217 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:36,217 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:36,217 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:36,218 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,218 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:36,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:36,218 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:36,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:36,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:36,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:36,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,219 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:36,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,220 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:36,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,221 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,221 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:36,221 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:36,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:36,222 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:36,222 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:36,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:36,222 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,222 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,222 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,222 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,222 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:36,222 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:36,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:36,222 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:36,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:36,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:36,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:36,301 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3df9cf7b9785edd3a0db773c7f6b886b 3/3 column families, dataSize=143.13 KB heapSize=395.92 KB 2023-04-28 14:57:36,301 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=A 2023-04-28 14:57:36,301 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:36,301 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=B 2023-04-28 14:57:36,301 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:36,301 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=C 2023-04-28 14:57:36,301 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:36,310 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:36,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48948 deadline: 1682693916308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,311 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:36,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693916309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,313 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:36,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693916310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,322 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:36,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:36,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:36,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:36,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:36,331 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:36,331 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:36,331 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:36,331 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:36,332 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,332 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,332 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,332 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:36,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:36,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:36,333 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,333 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:36,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:36,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,334 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:36,334 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:36,334 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:36,334 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:36,334 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:36,334 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:36,334 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:36,334 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:36,334 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:36,334 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:36,334 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:36,334 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:36,334 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:36,334 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:36,334 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,334 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:36,334 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:36,335 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,335 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:36,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:36,336 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,336 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:36,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:36,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:36,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:36,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:36,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@291732aa[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:36,337 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:36,337 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,338 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,338 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,338 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:36,338 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:36,338 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:36,338 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:36,339 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:36,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:36,339 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:36,339 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:36,339 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:36,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:36,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:36,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:36,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:36,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:36,341 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:36,341 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:36,341 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:36,341 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:36,341 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:36,342 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:36,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:36,342 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,342 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:36,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:36,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:36,343 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:36,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:36,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,344 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:36,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,345 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,345 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,345 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,345 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,345 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,345 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,345 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,345 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:36,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:36,345 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:36,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:36,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:36,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:36,346 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:36,346 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:36,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:36,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:36,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:36,345 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:36,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:36,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48948 deadline: 1682693916412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,414 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:36,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693916412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:36,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693916413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,446 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:36,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:36,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:36,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:36,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:36,449 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:36,449 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:36,449 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:36,449 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:36,449 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,449 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,450 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:36,450 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:36,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:36,450 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,450 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:36,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:36,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:36,451 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:36,451 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:36,451 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:36,451 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:36,451 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:36,451 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:36,451 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:36,451 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:36,451 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:36,451 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:36,451 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:36,451 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:36,451 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:36,451 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,451 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:36,451 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:36,452 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,452 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:36,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:36,453 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,453 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:36,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:36,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:36,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:36,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:36,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@117b8460[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:36,454 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:36,454 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,454 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,455 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,455 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:36,455 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:36,455 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:36,455 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:36,455 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,455 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:36,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:36,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:36,456 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:36,456 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:36,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:36,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:36,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:36,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:36,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:36,458 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:36,458 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:36,459 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:36,459 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:36,459 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:36,459 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,459 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:36,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,460 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:36,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:36,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:36,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:36,460 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:36,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:36,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,461 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:36,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,463 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:36,463 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:36,463 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:36,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:36,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:36,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:36,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:36,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:36,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:36,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:36,463 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,463 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,463 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,463 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:36,463 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:36,463 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,463 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,463 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,463 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,563 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:36,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:36,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:36,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:36,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:36,566 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:36,566 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:36,566 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:36,566 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:36,566 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,566 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,567 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,567 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:36,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:36,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:36,567 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,567 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:36,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:36,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,568 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:36,568 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:36,568 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:36,568 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:36,568 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:36,568 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:36,568 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:36,568 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:36,568 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:36,568 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:36,568 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:36,568 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:36,569 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:36,569 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:36,569 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,569 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:36,569 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:36,570 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,570 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:36,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:36,570 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,570 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:36,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:36,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:36,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:36,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:36,572 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:36,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4586e0bc[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:36,572 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,572 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,572 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:36,572 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:36,572 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,572 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:36,572 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:36,573 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:36,573 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,573 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:36,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:36,573 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:36,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:36,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:36,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:36,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:36,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:36,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:36,575 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:36,575 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:36,575 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:36,575 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:36,575 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:36,576 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,576 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:36,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:36,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,576 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:36,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:36,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:36,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:36,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,577 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:36,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,578 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:36,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,580 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,580 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:36,580 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,580 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:36,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:36,580 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:36,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:36,580 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:36,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:36,580 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:36,580 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:36,580 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:36,580 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,580 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,580 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:36,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:36,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:36,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48948 deadline: 1682693916614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:36,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693916615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:36,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693916617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,680 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:36,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:36,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:36,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:36,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:36,684 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:36,684 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:36,684 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:36,684 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:36,688 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,688 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,688 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,689 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:36,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:36,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:36,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,689 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,689 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:36,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:36,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:36,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:36,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:36,690 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:36,690 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:36,690 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:36,695 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:36,696 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:36,696 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:36,696 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:36,696 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:36,696 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:36,696 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:36,696 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:36,696 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,696 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:36,696 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:36,697 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,697 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:36,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:36,698 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,698 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:36,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:36,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:36,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:36,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:36,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43df61f[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:36,700 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:36,700 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,702 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,702 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:36,702 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:36,702 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:36,702 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,702 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:36,703 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,703 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:36,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:36,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:36,703 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:36,703 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:36,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:36,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:36,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:36,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:36,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:36,704 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:36,704 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:36,704 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:36,705 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:36,705 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:36,705 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:36,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:36,706 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,706 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:36,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:36,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:36,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:36,706 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:36,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,707 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:36,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,708 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,708 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:36,708 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:36,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:36,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:36,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:36,708 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,709 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,709 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,709 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,709 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,708 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:36,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:36,708 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:36,709 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:36,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:36,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:36,709 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:36,715 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304285b836af016074589815136e1560b1eca to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304285b836af016074589815136e1560b1eca 2023-04-28 14:57:36,731 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=279, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/7fc7ff42602c42e5923e10489fa0db83 2023-04-28 14:57:36,809 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:36,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:36,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:36,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:36,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:36,812 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:36,812 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:36,812 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:36,812 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:36,812 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,812 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,813 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,813 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:36,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:36,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:36,813 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,813 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:36,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:36,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,814 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:36,814 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:36,814 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:36,814 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:36,814 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:36,814 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:36,815 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:36,815 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:36,815 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:36,815 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:36,815 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:36,815 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:36,815 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:36,815 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:36,815 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,815 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:36,815 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:36,816 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,816 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:36,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:36,816 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,816 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:36,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:36,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:36,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:36,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:36,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@142f0650[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:36,817 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:36,817 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,818 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,818 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:36,818 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:36,818 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,818 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:36,818 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:36,819 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,819 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:36,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:36,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:36,819 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:36,819 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:36,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:36,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:36,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:36,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:36,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:36,821 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:36,822 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:36,822 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:36,822 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:36,822 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:36,823 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,823 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:36,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:36,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:36,824 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:36,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:36,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:36,824 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:36,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,826 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:36,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,827 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,827 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,827 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,827 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:36,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:36,827 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:36,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:36,827 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,827 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:36,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:36,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:36,827 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:36,828 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,828 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,828 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:36,828 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:36,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:36,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:36,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:36,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48948 deadline: 1682693916918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:36,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693916918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:36,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693916919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,928 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:36,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:36,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:36,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:36,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:36,931 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:36,931 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:36,931 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:36,931 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:36,931 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,931 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,932 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:36,932 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:36,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:36,932 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,932 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:36,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:36,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,933 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:36,933 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:36,933 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:36,933 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:36,933 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:36,933 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:36,933 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:36,933 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:36,933 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:36,933 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:36,933 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:36,934 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:36,934 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:36,934 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:36,934 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,934 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:36,934 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:36,934 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,934 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:36,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:36,935 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,935 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:36,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:36,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:36,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:36,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:36,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@593ad831[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:36,936 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:36,936 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,937 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,937 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:36,937 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:36,937 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:36,937 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,937 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:36,938 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,938 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:36,938 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:36,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:36,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:36,938 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:36,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:36,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:36,940 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:36,940 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:36,940 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:36,940 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:36,941 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:36,942 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,942 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:36,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:36,942 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:36,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,943 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:36,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:36,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:36,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:36,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,944 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:36,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:36,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,945 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,945 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:36,945 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,945 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:36,945 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:36,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:36,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:36,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:36,945 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:36,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:36,945 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:36,946 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,946 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:36,946 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:36,946 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:36,946 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:36,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:36,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:37,046 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:37,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:37,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:37,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:37,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:37,050 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:37,050 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:37,050 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:37,050 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:37,050 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,050 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,051 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,051 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:37,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:37,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:37,051 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,051 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:37,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:37,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,052 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:37,052 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:37,052 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:37,052 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:37,052 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:37,052 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:37,053 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:37,053 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:37,053 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:37,053 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:37,054 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:37,054 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:37,054 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:37,054 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:37,054 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,054 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:37,054 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:37,055 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,055 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:37,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:37,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:37,056 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,056 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:37,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:37,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:37,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:37,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bcb50aa[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:37,058 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:37,058 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,059 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,059 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,059 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:37,059 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:37,059 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:37,060 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:37,060 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:37,060 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:37,060 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:37,060 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:37,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:37,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:37,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:37,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:37,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:37,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:37,065 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:37,065 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:37,065 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:37,065 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:37,065 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:37,067 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,067 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:37,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,067 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:37,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:37,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:37,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:37,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:37,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,068 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:37,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:37,071 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,073 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:37,073 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:37,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:37,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:37,073 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,073 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,073 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:37,073 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:37,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:37,073 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,073 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,073 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,073 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,073 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:37,073 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:37,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:37,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:37,151 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/3ea8af070900425787b97f94726ce1a9 2023-04-28 14:57:37,174 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:37,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:37,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:37,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:37,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:37,176 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:37,176 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:37,176 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:37,177 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:37,178 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,178 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,178 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,178 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:37,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:37,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:37,179 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,179 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:37,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:37,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:37,180 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:37,180 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:37,180 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:37,180 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:37,180 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:37,180 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:37,180 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:37,180 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:37,180 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:37,180 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:37,180 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:37,180 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:37,180 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:37,180 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,180 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:37,180 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:37,181 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,181 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:37,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:37,181 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,181 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:37,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:37,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:37,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:37,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:37,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@402dd6c2[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:37,182 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:37,183 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,183 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,183 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:37,183 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,184 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:37,183 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:37,184 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:37,184 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,184 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:37,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:37,184 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:37,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:37,185 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:37,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:37,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:37,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:37,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:37,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:37,186 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:37,186 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:37,186 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:37,186 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:37,186 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:37,188 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,188 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:37,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:37,188 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:37,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:37,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:37,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:37,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,188 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:37,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,190 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:37,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,191 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:37,191 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,191 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:37,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:37,191 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,191 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:37,191 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:37,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:37,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:37,191 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,191 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,191 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,191 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,191 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:37,192 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:37,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:37,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:37,292 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:37,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:37,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:37,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:37,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:37,294 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:37,294 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:37,294 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:37,294 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:37,295 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,295 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,295 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:37,295 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:37,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:37,297 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,297 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:37,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:37,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,297 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:37,297 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:37,297 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:37,298 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:37,298 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:37,298 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:37,298 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:37,298 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:37,298 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:37,298 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:37,298 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:37,298 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:37,298 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:37,298 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:37,298 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,298 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:37,298 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:37,299 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,299 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:37,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:37,299 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,299 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:37,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:37,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:37,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:37,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:37,300 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:37,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c5c4511[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:37,300 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,301 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,301 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:37,301 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:37,301 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:37,301 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,301 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:37,301 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:37,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:37,302 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:37,302 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:37,302 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:37,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:37,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:37,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:37,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:37,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:37,303 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:37,304 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:37,304 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:37,304 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:37,304 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:37,304 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,304 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,305 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:37,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:37,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:37,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:37,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:37,305 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:37,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:37,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,306 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:37,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,307 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,307 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:37,307 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,307 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:37,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:37,307 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:37,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:37,307 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:37,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:37,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:37,308 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:37,307 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,308 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:37,308 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,308 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:37,308 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:37,408 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:37,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:37,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:37,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:37,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:37,414 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:37,414 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:37,414 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:37,414 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:37,415 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,415 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,415 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:37,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:37,415 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:37,416 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,416 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:37,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:37,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,417 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:37,422 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:37,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:37,422 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:37,422 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:37,422 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:37,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:37,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693917420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:37,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48948 deadline: 1682693917422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:37,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693917425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,437 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:37,437 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:37,437 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:37,437 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:37,437 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:37,437 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:37,438 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:37,438 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:37,438 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,438 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:37,438 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:37,439 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,439 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:37,439 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:37,439 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:37,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:37,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:37,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2328ebfb[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:37,441 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:37,441 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,441 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,442 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:37,442 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:37,442 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:37,442 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,442 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:37,443 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,443 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:37,443 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:37,443 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:37,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:37,445 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:37,445 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:37,445 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:37,445 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:37,445 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:37,447 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,447 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:37,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:37,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,448 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:37,449 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:37,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,450 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:37,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,453 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:37,453 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:37,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:37,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:37,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:37,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:37,454 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,454 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,454 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,454 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:37,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:37,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:37,454 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,454 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:37,454 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,455 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:37,455 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,455 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,555 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:37,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:37,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:37,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:37,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:37,557 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:37,557 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:37,557 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:37,558 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:37,558 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,558 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,559 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:37,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:37,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:37,559 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,559 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,559 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:37,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:37,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,560 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:37,560 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:37,560 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:37,560 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:37,560 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:37,560 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:37,561 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:37,561 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:37,561 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:37,561 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:37,561 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:37,561 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:37,561 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:37,561 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:37,561 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,561 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:37,561 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:37,561 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/9d3fc29e05b547c7907ea5e8b457eee6 2023-04-28 14:57:37,562 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:37,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:37,562 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:37,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:37,563 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,563 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:37,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:37,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:37,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5217b134[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:37,565 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:37,565 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,565 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,566 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:37,566 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:37,566 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:37,566 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,566 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:37,567 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/7fc7ff42602c42e5923e10489fa0db83 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/7fc7ff42602c42e5923e10489fa0db83 2023-04-28 14:57:37,567 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:37,567 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,567 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:37,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:37,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:37,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:37,567 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:37,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:37,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:37,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:37,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:37,572 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:37,572 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:37,572 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:37,572 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:37,572 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:37,572 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/7fc7ff42602c42e5923e10489fa0db83, entries=200, sequenceid=279, filesize=32.3 K 2023-04-28 14:57:37,573 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:37,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:37,574 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,574 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:37,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,574 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:37,574 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/3ea8af070900425787b97f94726ce1a9 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/3ea8af070900425787b97f94726ce1a9 2023-04-28 14:57:37,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:37,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:37,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:37,575 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:37,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,579 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:37,579 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:37,579 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,579 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:37,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:37,579 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,579 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:37,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:37,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:37,579 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,579 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,579 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,579 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,579 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:37,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:37,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:37,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:37,579 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:37,580 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/3ea8af070900425787b97f94726ce1a9, entries=150, sequenceid=279, filesize=11.9 K 2023-04-28 14:57:37,581 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/9d3fc29e05b547c7907ea5e8b457eee6 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/9d3fc29e05b547c7907ea5e8b457eee6 2023-04-28 14:57:37,586 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/9d3fc29e05b547c7907ea5e8b457eee6, entries=150, sequenceid=279, filesize=11.9 K 2023-04-28 14:57:37,586 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~143.13 KB/146560, heapSize ~395.88 KB/405376, currentSize=40.25 KB/41220 for 3df9cf7b9785edd3a0db773c7f6b886b in 1285ms, sequenceid=279, compaction requested=true 2023-04-28 14:57:37,587 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:37,587 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:57:37,587 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:37,587 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:57:37,587 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:37,587 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:57:37,589 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104050 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:57:37,589 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): 3df9cf7b9785edd3a0db773c7f6b886b/A is initiating minor compaction (all files) 2023-04-28 14:57:37,589 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3df9cf7b9785edd3a0db773c7f6b886b/A in TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:37,589 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/25fc7e9715ad4320b85f7011c6e1b85d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/396612d922734853942f822e06397ed1, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/00d850e9e50c49cf8b0b7a1b524f8158, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/7fc7ff42602c42e5923e10489fa0db83] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp, totalSize=101.6 K 2023-04-28 14:57:37,589 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46174 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:57:37,589 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 3df9cf7b9785edd3a0db773c7f6b886b/B is initiating minor compaction (all files) 2023-04-28 14:57:37,589 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3df9cf7b9785edd3a0db773c7f6b886b/B in TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:37,589 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/0e47d7e42d1a4f0390880d8a800dcd16, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/a3095a4faa0c43d68cff36a4545ae1ad, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/f6c9de97a9504b49b28be51d5a67cd16, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/3ea8af070900425787b97f94726ce1a9] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp, totalSize=45.1 K 2023-04-28 14:57:37,590 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 25fc7e9715ad4320b85f7011c6e1b85d, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682693853253 2023-04-28 14:57:37,590 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 396612d922734853942f822e06397ed1, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1682693853450 2023-04-28 14:57:37,590 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 0e47d7e42d1a4f0390880d8a800dcd16, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682693853253 2023-04-28 14:57:37,591 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 00d850e9e50c49cf8b0b7a1b524f8158, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1682693854518 2023-04-28 14:57:37,591 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting a3095a4faa0c43d68cff36a4545ae1ad, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1682693853450 2023-04-28 14:57:37,591 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 7fc7ff42602c42e5923e10489fa0db83, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1682693855680 2023-04-28 14:57:37,591 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting f6c9de97a9504b49b28be51d5a67cd16, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1682693854518 2023-04-28 14:57:37,591 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 3ea8af070900425787b97f94726ce1a9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1682693855687 2023-04-28 14:57:37,601 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 3df9cf7b9785edd3a0db773c7f6b886b#B#compaction#141 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:37,624 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3df9cf7b9785edd3a0db773c7f6b886b#A#compaction#142 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:37,680 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:37,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:37,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:37,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:37,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:37,682 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:37,682 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:37,682 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:37,682 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:37,683 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,683 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,683 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:37,683 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:37,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:37,684 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,684 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:37,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:37,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,688 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:37,688 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:37,688 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:37,688 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:37,688 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:37,688 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:37,688 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:37,689 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:37,689 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:37,689 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3df9cf7b9785edd3a0db773c7f6b886b 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-28 14:57:37,689 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=A 2023-04-28 14:57:37,689 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:37,689 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=B 2023-04-28 14:57:37,689 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:37,690 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=C 2023-04-28 14:57:37,690 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:37,716 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304280c0fc21912ed4c7da6fe2a4cb099ff4f to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304280c0fc21912ed4c7da6fe2a4cb099ff4f 2023-04-28 14:57:37,728 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=288, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/bcc4599df23b4ecf892ae44c2be548cd 2023-04-28 14:57:37,760 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/3875048f3b594425bef5faaeffd7a55a 2023-04-28 14:57:37,789 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/8ac89a44d78940758776143061eff010 2023-04-28 14:57:37,795 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/bcc4599df23b4ecf892ae44c2be548cd as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/bcc4599df23b4ecf892ae44c2be548cd 2023-04-28 14:57:37,800 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/bcc4599df23b4ecf892ae44c2be548cd, entries=150, sequenceid=288, filesize=25.4 K 2023-04-28 14:57:37,800 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/3875048f3b594425bef5faaeffd7a55a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/3875048f3b594425bef5faaeffd7a55a 2023-04-28 14:57:37,804 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/3875048f3b594425bef5faaeffd7a55a, entries=150, sequenceid=288, filesize=11.9 K 2023-04-28 14:57:37,808 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/8ac89a44d78940758776143061eff010 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/8ac89a44d78940758776143061eff010 2023-04-28 14:57:37,814 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/8ac89a44d78940758776143061eff010, entries=150, sequenceid=288, filesize=11.9 K 2023-04-28 14:57:37,815 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for 3df9cf7b9785edd3a0db773c7f6b886b in 125ms, sequenceid=288, compaction requested=true 2023-04-28 14:57:37,815 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:37,815 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:37,815 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:37,815 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:37,815 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,815 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:37,815 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:37,816 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,816 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:37,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:37,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:37,816 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,816 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:37,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:37,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:37,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:37,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ca2f672[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:37,818 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:37,818 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,818 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,819 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:37,819 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:37,819 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:37,819 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,819 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:37,819 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:37,819 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,819 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:37,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:37,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:37,820 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:37,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:37,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:37,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:37,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:37,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:37,822 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:37,822 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:37,822 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:37,822 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:37,822 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:37,824 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,824 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:37,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,824 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:37,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:37,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:37,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:37,827 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:37,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:37,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,828 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:37,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,831 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,831 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,831 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,831 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,831 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,832 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,832 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,832 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:37,832 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:37,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:37,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:37,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:37,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:37,832 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:37,832 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:37,832 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:37,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:37,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:37,932 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:37,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:37,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:37,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:37,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:37,936 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:37,936 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:37,936 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:37,936 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:37,936 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,936 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,938 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,938 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:37,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:37,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:37,940 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,940 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:37,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:37,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,942 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:37,942 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:37,942 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:37,942 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:37,942 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:37,943 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:37,943 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:37,943 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:37,943 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:37,943 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:37,944 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:37,944 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:37,944 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:37,944 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:37,944 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,944 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:37,944 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:37,945 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,945 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:37,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:37,945 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,945 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:37,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:37,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:37,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:37,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:37,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3218503a[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:37,947 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:37,947 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,948 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,948 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:37,950 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,950 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:37,950 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:37,951 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:37,953 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,953 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:37,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,953 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:37,953 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:37,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:37,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:37,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:37,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:37,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:37,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:37,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:37,955 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:37,955 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:37,955 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:37,955 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:37,955 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:37,957 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,957 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:37,957 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:37,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:37,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:37,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:37,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:37,958 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:37,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,959 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:37,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:37,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,961 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:37,961 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:37,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:37,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:37,961 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:37,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:37,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:37,961 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,961 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,961 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:37,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:37,961 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:37,961 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:37,961 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:37,961 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:37,961 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:37,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:37,961 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:38,023 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/1b6adf53932a4740b76bef50ae4a4381 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/1b6adf53932a4740b76bef50ae4a4381 2023-04-28 14:57:38,034 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3df9cf7b9785edd3a0db773c7f6b886b/B of 3df9cf7b9785edd3a0db773c7f6b886b into 1b6adf53932a4740b76bef50ae4a4381(size=12.3 K), total size for store is 24.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:38,034 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:38,034 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b., storeName=3df9cf7b9785edd3a0db773c7f6b886b/B, priority=12, startTime=1682693857587; duration=0sec 2023-04-28 14:57:38,034 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:57:38,034 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 14:57:38,036 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58360 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 14:57:38,036 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 3df9cf7b9785edd3a0db773c7f6b886b/C is initiating minor compaction (all files) 2023-04-28 14:57:38,036 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3df9cf7b9785edd3a0db773c7f6b886b/C in TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:38,036 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/9a8f2e00703d4fd9b6a8c30d5ca4de8e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/1c02482d3f744352b57022433de4edea, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/c6d383d37a294f4886e897fc6e79b68b, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/9d3fc29e05b547c7907ea5e8b457eee6, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/8ac89a44d78940758776143061eff010] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp, totalSize=57.0 K 2023-04-28 14:57:38,037 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 9a8f2e00703d4fd9b6a8c30d5ca4de8e, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682693853253 2023-04-28 14:57:38,038 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 1c02482d3f744352b57022433de4edea, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1682693853450 2023-04-28 14:57:38,038 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting c6d383d37a294f4886e897fc6e79b68b, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1682693854518 2023-04-28 14:57:38,039 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 9d3fc29e05b547c7907ea5e8b457eee6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1682693855687 2023-04-28 14:57:38,039 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 8ac89a44d78940758776143061eff010, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1682693856307 2023-04-28 14:57:38,049 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 3df9cf7b9785edd3a0db773c7f6b886b#C#compaction#146 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:38,062 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:38,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:38,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:38,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:38,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:38,067 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:38,067 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:38,068 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:38,068 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:38,068 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:38,068 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:38,069 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:38,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:38,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:38,070 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:38,070 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,070 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:38,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:38,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:38,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:38,070 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:38,074 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:38,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:38,074 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:38,074 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:38,074 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:38,078 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:38,078 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:38,078 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/7e0a62e3a22b4a2e81238a9683b94f34 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/7e0a62e3a22b4a2e81238a9683b94f34 2023-04-28 14:57:38,078 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:38,082 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:38,083 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:38,083 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:38,083 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:38,083 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:38,083 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,083 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:38,083 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:38,085 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:38,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:38,085 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:38,086 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:38,086 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:38,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:38,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:38,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:38,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:38,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:38,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@568b12b1[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:38,088 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:38,088 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:38,088 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3df9cf7b9785edd3a0db773c7f6b886b/C of 3df9cf7b9785edd3a0db773c7f6b886b into 7e0a62e3a22b4a2e81238a9683b94f34(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:38,088 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:38,089 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b., storeName=3df9cf7b9785edd3a0db773c7f6b886b/C, priority=11, startTime=1682693857587; duration=0sec 2023-04-28 14:57:38,089 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:38,089 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:38,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:38,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:38,089 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:38,089 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:38,089 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:38,089 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,089 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:38,090 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,090 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:38,090 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:38,090 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:38,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:38,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:38,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:38,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:38,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:38,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:38,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:38,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:38,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:38,093 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:38,093 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:38,093 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:38,093 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:38,093 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:38,095 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:38,095 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:38,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:38,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:38,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:38,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:38,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:38,095 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:38,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:38,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:38,096 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:38,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:38,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:38,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:38,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:38,099 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:38,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:38,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,101 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,101 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:38,101 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:38,101 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:38,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:38,101 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:38,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:38,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:38,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:38,101 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:38,101 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,101 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:38,101 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:38,102 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:38,101 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:38,102 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:38,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:38,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:38,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:38,202 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:38,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:38,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:38,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:38,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:38,205 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:38,205 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:38,205 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:38,205 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:38,206 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:38,206 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:38,207 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,207 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:38,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:38,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:38,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:38,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:38,208 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,208 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:38,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:38,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:38,208 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:38,208 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:38,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:38,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:38,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:38,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:38,210 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:38,210 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:38,210 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:38,210 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:38,210 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:38,210 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:38,210 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:38,211 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:38,211 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,211 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:38,211 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:38,212 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,212 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:38,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:38,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:38,212 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:38,212 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:38,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:38,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:38,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:38,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:38,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:38,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@397a8745[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:38,214 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:38,214 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:38,215 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:38,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:38,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:38,216 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:38,216 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:38,216 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:38,216 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,216 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:38,217 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,217 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:38,217 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:38,217 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:38,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:38,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:38,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:38,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:38,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:38,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:38,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:38,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:38,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:38,221 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:38,221 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:38,221 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:38,221 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:38,221 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:38,222 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:38,222 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:38,222 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:38,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:38,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:38,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:38,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:38,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:38,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:38,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:38,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:38,223 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:38,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:38,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:38,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:38,224 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:38,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:38,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,228 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:38,228 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:38,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:38,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:38,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:38,228 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:38,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:38,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:38,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:38,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:38,229 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,229 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:38,229 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:38,229 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,229 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:38,230 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:38,230 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:38,230 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:38,230 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:38,330 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:38,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:38,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:38,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:38,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:38,333 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:38,333 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:38,333 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:38,333 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:38,334 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:38,334 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:38,334 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,334 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:38,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:38,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:38,335 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,335 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:38,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:38,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:38,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:38,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:38,335 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:38,336 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:38,336 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:38,336 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:38,336 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:38,336 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:38,336 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:38,336 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:38,336 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:38,336 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:38,336 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:38,336 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:38,336 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:38,336 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:38,337 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,337 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:38,337 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:38,338 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,338 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:38,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:38,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:38,338 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:38,338 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:38,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:38,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:38,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:38,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:38,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:38,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5280e77e[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:38,342 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:38,342 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:38,342 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:38,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:38,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:38,343 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:38,343 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:38,343 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:38,343 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,343 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:38,344 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:38,344 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:38,344 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,344 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:38,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:38,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:38,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:38,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:38,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:38,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:38,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:38,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:38,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:38,347 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:38,347 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:38,347 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:38,347 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:38,347 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:38,348 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:38,348 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:38,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:38,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:38,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:38,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:38,348 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:38,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:38,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:38,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:38,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:38,349 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:38,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:38,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:38,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:38,350 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:38,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:38,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,352 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:38,352 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:38,352 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,352 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:38,352 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:38,352 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,352 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:38,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:38,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:38,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:38,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:38,352 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:38,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:38,353 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:38,353 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:38,353 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:38,353 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:38,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:38,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:38,453 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:38,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:38,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:38,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:38,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:38,462 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:38,462 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:38,462 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:38,462 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:38,463 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:38,463 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:38,463 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:38,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:38,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:38,464 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:38,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:38,465 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,465 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:38,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:38,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:38,465 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:38,466 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:38,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:38,466 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:38,466 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:38,466 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:38,478 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:38,478 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:38,478 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:38,478 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:38,478 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3df9cf7b9785edd3a0db773c7f6b886b 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-28 14:57:38,478 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=A 2023-04-28 14:57:38,479 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:38,479 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=B 2023-04-28 14:57:38,479 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:38,479 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=C 2023-04-28 14:57:38,479 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:38,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:38,494 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:38,514 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/5cfe200f93ec46cebe08ba5fb7b4e7e1 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/5cfe200f93ec46cebe08ba5fb7b4e7e1 2023-04-28 14:57:38,520 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3df9cf7b9785edd3a0db773c7f6b886b/A of 3df9cf7b9785edd3a0db773c7f6b886b into 5cfe200f93ec46cebe08ba5fb7b4e7e1(size=25.8 K), total size for store is 51.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:38,520 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:38,520 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b., storeName=3df9cf7b9785edd3a0db773c7f6b886b/A, priority=12, startTime=1682693857587; duration=0sec 2023-04-28 14:57:38,520 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:38,529 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428aef9a07c4dd54475b20b5f2f8f844e74 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428aef9a07c4dd54475b20b5f2f8f844e74 2023-04-28 14:57:38,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693858527, max timestamp=1682693858534Num uniques 859; 2023-04-28 14:57:38,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:57:38,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693858527, max timestamp=1682693858534Num uniques 859; 2023-04-28 14:57:38,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693858527, max timestamp=1682693858534Num uniques 859; 2023-04-28 14:57:38,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:57:38,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:57:38,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:38,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693918562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,566 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682693858554, max timestamp=1682693858563Num uniques 1068; 2023-04-28 14:57:38,566 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-28 14:57:38,566 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682693858554, max timestamp=1682693858563Num uniques 1068; 2023-04-28 14:57:38,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-28 14:57:38,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682693858554, max timestamp=1682693858563Num uniques 1068; 2023-04-28 14:57:38,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-28 14:57:38,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1682693858582, max timestamp=1682693858591Num uniques 1127; 2023-04-28 14:57:38,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1682693858582, max timestamp=1682693858591Num uniques 1127; 2023-04-28 14:57:38,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-04-28 14:57:38,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-04-28 14:57:38,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1682693858582, max timestamp=1682693858591Num uniques 1127; 2023-04-28 14:57:38,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-04-28 14:57:38,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:38,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48998 deadline: 1682693918623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,627 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682693858616, max timestamp=1682693858624Num uniques 1186; 2023-04-28 14:57:38,627 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=8516, suffix heap size=151312,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=186 2023-04-28 14:57:38,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682693858616, max timestamp=1682693858624Num uniques 1186; 2023-04-28 14:57:38,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=6870, suffix heap size=151312,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=150 2023-04-28 14:57:38,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682693858616, max timestamp=1682693858624Num uniques 1186; 2023-04-28 14:57:38,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=6870, suffix heap size=151312,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=150 2023-04-28 14:57:38,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1682693858637, max timestamp=1682693858657Num uniques 1045; 2023-04-28 14:57:38,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=15800, suffix heap size=136648,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=345 2023-04-28 14:57:38,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1682693858637, max timestamp=1682693858657Num uniques 1009; 2023-04-28 14:57:38,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=15800, suffix heap size=132904,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=345 2023-04-28 14:57:38,661 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1682693858637, max timestamp=1682693858657Num uniques 1009; 2023-04-28 14:57:38,661 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=15800, suffix heap size=132904,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=345 2023-04-28 14:57:38,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:38,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693918680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:38,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48948 deadline: 1682693918681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,685 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:38,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48932 deadline: 1682693918682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:38,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693918683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=404, cellSize=18500, totalHeapSize=42256, min timestamp=1682693858673, max timestamp=1682693858682Num uniques 1204; 2023-04-28 14:57:38,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=18500, suffix heap size=153184,new segment heap size=42256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=404 2023-04-28 14:57:38,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=404, cellSize=18500, totalHeapSize=42256, min timestamp=1682693858673, max timestamp=1682693858682Num uniques 1204; 2023-04-28 14:57:38,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=18500, suffix heap size=153184,new segment heap size=42256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=404 2023-04-28 14:57:38,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=404, cellSize=18500, totalHeapSize=42256, min timestamp=1682693858673, max timestamp=1682693858682Num uniques 1204; 2023-04-28 14:57:38,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=18500, suffix heap size=153184,new segment heap size=42256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=404 2023-04-28 14:57:38,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:38,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48998 deadline: 1682693918778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:38,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693918784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,793 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:38,793 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:38,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48948 deadline: 1682693918788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693918788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,793 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:38,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48932 deadline: 1682693918788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:38,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48998 deadline: 1682693918882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,940 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=301, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/e6d10f7ecb2e48458de94e5867c5771c 2023-04-28 14:57:38,949 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/f367eb15728c48de9ff12866c265b5c7 2023-04-28 14:57:38,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:38,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693918986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:38,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693918993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:38,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48948 deadline: 1682693918995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:38,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:38,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48932 deadline: 1682693918995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,088 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:39,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48998 deadline: 1682693919086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,294 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:39,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693919291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:39,296 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:39,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693919296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48948 deadline: 1682693919296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,300 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:39,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48932 deadline: 1682693919297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,367 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/313facf96c8c4f62891cf6eafe5720af 2023-04-28 14:57:39,371 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/e6d10f7ecb2e48458de94e5867c5771c as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/e6d10f7ecb2e48458de94e5867c5771c 2023-04-28 14:57:39,375 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/e6d10f7ecb2e48458de94e5867c5771c, entries=200, sequenceid=301, filesize=32.3 K 2023-04-28 14:57:39,376 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/f367eb15728c48de9ff12866c265b5c7 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/f367eb15728c48de9ff12866c265b5c7 2023-04-28 14:57:39,379 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/f367eb15728c48de9ff12866c265b5c7, entries=100, sequenceid=301, filesize=9.5 K 2023-04-28 14:57:39,380 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/313facf96c8c4f62891cf6eafe5720af as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/313facf96c8c4f62891cf6eafe5720af 2023-04-28 14:57:39,384 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/313facf96c8c4f62891cf6eafe5720af, entries=100, sequenceid=301, filesize=9.5 K 2023-04-28 14:57:39,385 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=161.02 KB/164880 for 3df9cf7b9785edd3a0db773c7f6b886b in 906ms, sequenceid=301, compaction requested=true 2023-04-28 14:57:39,385 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:39,385 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:39,385 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:39,385 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:39,385 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,385 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:39,385 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:39,386 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,386 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:39,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:39,386 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,386 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:39,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:39,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:39,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:39,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:39,387 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:39,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22e9f494[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:39,387 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,388 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,388 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:39,388 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:39,388 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:39,389 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:39,389 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:39,389 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:39,389 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,390 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:39,390 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:39,390 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:39,390 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:39,390 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:39,390 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:39,390 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:39,391 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,391 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:39,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:39,391 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:39,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:39,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:39,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:39,392 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:39,392 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3df9cf7b9785edd3a0db773c7f6b886b 3/3 column families, dataSize=167.72 KB heapSize=461.63 KB 2023-04-28 14:57:39,392 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=A 2023-04-28 14:57:39,392 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:39,392 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=B 2023-04-28 14:57:39,392 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:39,392 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=C 2023-04-28 14:57:39,392 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:39,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:39,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,393 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:39,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,396 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,396 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:39,396 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:39,396 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:39,396 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:39,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:39,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:39,396 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:39,396 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,396 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:39,396 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:39,396 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:39,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:39,396 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:39,397 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:39,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:39,401 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:39,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48998 deadline: 1682693919399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,496 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:39,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:39,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:39,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:39,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:39,499 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:39,499 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:39,499 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:39,499 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:39,500 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,500 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:39,500 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:39,500 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:39,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:39,501 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:39,501 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:39,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:39,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:39,501 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:39,501 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:39,501 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:39,501 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:39,501 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:39,501 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:39,502 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:39,502 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:39,502 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:39,502 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:39,502 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:39,502 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:39,502 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:39,502 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:39,502 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,502 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:39,502 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:39,503 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,503 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:39,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:39,503 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,503 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:39,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:39,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:39,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:39,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48998 deadline: 1682693919502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:39,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:39,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c7964e2[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:39,504 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:39,504 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,505 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,506 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:39,506 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:39,506 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:39,506 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,506 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:39,507 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,507 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:39,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:39,507 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:39,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:39,507 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:39,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:39,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:39,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:39,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:39,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:39,509 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:39,509 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:39,509 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:39,509 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:39,509 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:39,510 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,510 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:39,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:39,510 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:39,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,510 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:39,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:39,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:39,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:39,511 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:39,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,512 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:39,513 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:39,513 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:39,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:39,513 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:39,513 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:39,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:39,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:39,513 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:39,513 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,513 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,513 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:39,513 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:39,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:39,513 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,513 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:39,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:39,613 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:39,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:39,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:39,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:39,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:39,616 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:39,616 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:39,616 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:39,616 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:39,616 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,616 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:39,618 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,618 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:39,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:39,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:39,618 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,618 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:39,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:39,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:39,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:39,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:39,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:39,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:39,619 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:39,619 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:39,619 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:39,619 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:39,619 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:39,620 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:39,620 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:39,620 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:39,620 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:39,620 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:39,620 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:39,620 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,620 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:39,620 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:39,621 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,621 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:39,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:39,622 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,622 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:39,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:39,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:39,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:39,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:39,623 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:39,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41548e87[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:39,623 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,623 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,623 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:39,623 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,623 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:39,623 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:39,624 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:39,624 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,624 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:39,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:39,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:39,624 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:39,624 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:39,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:39,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:39,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:39,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:39,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:39,626 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:39,626 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:39,626 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:39,626 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:39,626 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:39,627 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:39,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:39,627 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,627 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:39,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:39,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:39,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:39,628 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:39,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,629 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:39,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,630 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,630 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:39,630 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:39,630 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,630 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:39,630 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:39,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:39,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:39,630 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:39,630 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:39,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:39,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:39,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:39,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:39,631 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,631 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,631 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:39,631 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:39,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:39,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48998 deadline: 1682693919705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,731 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:39,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:39,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:39,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:39,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:39,735 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:39,735 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:39,735 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:39,735 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:39,735 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,735 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:39,736 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:39,736 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:39,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:39,736 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,736 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:39,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:39,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:39,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:39,743 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:39,745 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:39,745 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:39,745 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:39,745 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:39,745 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:39,745 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:39,745 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:39,745 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:39,745 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:39,745 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:39,745 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:39,745 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:39,745 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:39,745 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,745 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:39,745 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:39,746 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,746 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:39,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:39,746 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,746 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:39,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:39,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:39,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:39,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:39,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14ba2ac4[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:39,748 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:39,748 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,748 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,748 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:39,748 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:39,748 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:39,749 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,749 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:39,749 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,749 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:39,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:39,749 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:39,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:39,749 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:39,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:39,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:39,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:39,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:39,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:39,751 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:39,751 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:39,751 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:39,751 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:39,751 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:39,752 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,752 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:39,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:39,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,752 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:39,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:39,753 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:39,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:39,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:39,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,754 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:39,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,755 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,755 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:39,755 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:39,755 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:39,755 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:39,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:39,755 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:39,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:39,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:39,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:39,755 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:39,755 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:39,756 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:39,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:39,756 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,756 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:39,756 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:39,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48948 deadline: 1682693919797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:39,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48944 deadline: 1682693919797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,801 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:39,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693919801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:39,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48932 deadline: 1682693919803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,808 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428cd70b5dbff964839aa5ed31bd7c13b29 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428cd70b5dbff964839aa5ed31bd7c13b29 2023-04-28 14:57:39,856 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:39,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:39,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:39,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:39,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:39,859 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:39,859 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:39,859 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:39,859 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:39,859 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,859 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:39,860 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:39,860 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:39,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:39,860 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,860 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:39,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:39,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:39,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:39,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:39,861 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:39,861 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:39,861 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:39,861 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:39,861 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:39,861 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:39,861 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:39,861 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:39,861 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:39,861 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:39,861 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:39,862 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:39,862 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:39,862 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,862 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:39,862 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:39,862 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,862 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:39,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:39,863 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,863 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:39,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:39,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:39,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:39,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:39,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@791a3193[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:39,864 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:39,864 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,864 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,865 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:39,865 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:39,865 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:39,865 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,865 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:39,865 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:39,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:39,865 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:39,865 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:39,865 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:39,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:39,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:39,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:39,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:39,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:39,868 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:39,868 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:39,868 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:39,868 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:39,868 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:39,869 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,869 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:39,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:39,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,869 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:39,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:39,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:39,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:39,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:39,871 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:39,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,872 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,873 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,873 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:39,873 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:39,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:39,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:39,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:39,873 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:39,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:39,873 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:39,873 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:39,873 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,873 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,874 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,874 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,874 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:39,874 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:39,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:39,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:39,979 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:39,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:39,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:39,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:39,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:39,983 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:39,983 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:39,983 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:39,983 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:39,983 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,983 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:39,984 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:39,984 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:39,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:39,984 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,984 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:39,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:39,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:39,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:39,985 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:39,985 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:39,985 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:39,985 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:39,985 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:39,985 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:39,985 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:39,985 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:39,985 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:39,985 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:39,985 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:39,985 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:39,985 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:39,985 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,986 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:39,986 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:39,986 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,986 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:39,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:39,987 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,987 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:39,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:39,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:39,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:39,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:39,988 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:39,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ea30426[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:39,988 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,989 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,989 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:39,989 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:39,989 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:39,989 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,989 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:39,990 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,990 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:39,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:39,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:39,990 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:39,990 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:39,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:39,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:39,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:39,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:39,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:39,992 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:39,992 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:39,992 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:39,992 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:39,992 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:39,993 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,993 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,993 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:39,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:39,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:39,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:39,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:39,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:39,994 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:39,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,995 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:39,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:39,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,997 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:39,997 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:39,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:39,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:39,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:39,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:39,997 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:39,997 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,997 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:39,997 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:39,997 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:39,997 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:39,997 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:39,997 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,997 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:39,997 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:39,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:39,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:40,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:40,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48998 deadline: 1682693920009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,097 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:40,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:40,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:40,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:40,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:40,100 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:40,100 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:40,100 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:40,100 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:40,101 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,101 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:40,101 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,101 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:40,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:40,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:40,101 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,101 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:40,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:40,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:40,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:40,102 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:40,102 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:40,102 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:40,102 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:40,102 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:40,102 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:40,102 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:40,102 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:40,102 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:40,103 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:40,103 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:40,103 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:40,103 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:40,103 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:40,103 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,103 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:40,103 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:40,103 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,103 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:40,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:40,104 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,104 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:40,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:40,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:40,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:40,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:40,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:40,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b6691c3[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:40,105 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:40,105 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,105 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,105 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:40,105 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,106 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:40,105 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:40,106 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:40,106 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,106 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:40,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,106 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:40,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:40,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:40,106 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:40,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:40,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:40,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:40,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:40,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:40,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:40,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:40,108 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:40,108 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:40,108 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:40,108 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:40,108 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:40,109 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,109 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:40,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,109 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:40,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:40,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:40,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:40,110 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:40,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:40,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:40,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:40,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,111 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:40,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:40,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,112 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,112 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:40,112 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:40,112 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,112 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,112 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,112 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:40,112 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:40,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:40,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:40,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:40,112 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,112 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:40,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:40,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:40,113 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:40,113 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:40,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:40,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:40,213 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:40,213 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=425, memsize=55.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/e4ca72a0b0d14de68efbad79d12f4d45 2023-04-28 14:57:40,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:40,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:40,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:40,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:40,216 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:40,216 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:40,216 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:40,216 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:40,216 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,216 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:40,217 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,217 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:40,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:40,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:40,217 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,217 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:40,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:40,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:40,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:40,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:40,218 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:40,218 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:40,218 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:40,218 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:40,218 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:40,218 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:40,218 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:40,218 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:40,218 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:40,218 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:40,218 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:40,218 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:40,218 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:40,218 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,218 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:40,218 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:40,220 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,220 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:40,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:40,221 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,221 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:40,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:40,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:40,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:40,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:40,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:40,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e4f6861[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:40,222 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:40,222 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,223 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,223 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:40,223 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:40,223 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,223 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:40,223 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:40,224 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,224 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:40,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:40,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:40,224 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:40,224 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:40,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:40,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:40,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:40,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:40,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:40,226 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/1fbe0a5fed2b4e6b9b6a50a3f39daac7 2023-04-28 14:57:40,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:40,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:40,226 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:40,227 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:40,227 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:40,227 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:40,227 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:40,228 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,228 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:40,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:40,228 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:40,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:40,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:40,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:40,229 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:40,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:40,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:40,230 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,233 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:40,233 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:40,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:40,233 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:40,233 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:40,233 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:40,233 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,233 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,233 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:40,233 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:40,233 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,233 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:40,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:40,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:40,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:40,333 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:40,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:40,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:40,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:40,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:40,336 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:40,336 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:40,336 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:40,336 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:40,337 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,337 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:40,338 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:40,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:40,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:40,338 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:40,338 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,338 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:40,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:40,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:40,339 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:40,339 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:40,339 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:40,339 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:40,339 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:40,339 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:40,342 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:40,342 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:40,342 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:40,342 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:40,342 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:40,342 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:40,342 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:40,342 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:40,342 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,342 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:40,342 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:40,343 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,343 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,343 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:40,344 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:40,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:40,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:40,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:40,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:40,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:40,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:40,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16f15092[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:40,345 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:40,345 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,346 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,346 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:40,346 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,346 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:40,346 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:40,346 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:40,347 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,347 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:40,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:40,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:40,347 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:40,347 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:40,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:40,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:40,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:40,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:40,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:40,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:40,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:40,349 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:40,350 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:40,350 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:40,350 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:40,350 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:40,350 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:40,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:40,351 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,351 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:40,351 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:40,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:40,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:40,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:40,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:40,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:40,353 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:40,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:40,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,354 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,355 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:40,355 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:40,355 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:40,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:40,355 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:40,355 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:40,355 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:40,355 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,355 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:40,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:40,355 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:40,355 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:40,355 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:40,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:40,455 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:40,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:40,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:40,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:40,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:40,458 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:40,458 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:40,458 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:40,458 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:40,458 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,458 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:40,459 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,459 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:40,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:40,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:40,459 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,459 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:40,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:40,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:40,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:40,460 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:40,460 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:40,460 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:40,460 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:40,460 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:40,460 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:40,460 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:40,460 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:40,460 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:40,460 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:40,460 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:40,460 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:40,460 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:40,460 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:40,460 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,460 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:40,460 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:40,461 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,461 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:40,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:40,462 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,462 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:40,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:40,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:40,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:40,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:40,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:40,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1075e45e[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:40,463 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:40,463 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,463 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,463 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:40,463 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:40,463 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:40,463 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,464 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:40,464 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:40,464 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:40,464 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:40,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:40,464 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:40,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:40,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:40,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:40,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:40,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:40,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:40,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:40,466 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:40,466 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:40,466 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:40,466 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:40,466 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:40,467 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,467 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,467 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:40,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:40,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:40,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:40,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:40,468 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:40,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:40,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:40,469 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:40,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:40,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:40,471 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,471 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:40,471 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:40,471 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,471 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:40,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:40,471 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:40,471 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:40,471 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:40,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:40,471 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:40,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:40,471 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,471 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:40,471 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:40,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:40,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:40,513 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:40,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48998 deadline: 1682693920513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,572 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:40,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:40,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:40,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:40,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:40,574 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:40,574 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:40,574 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:40,574 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:40,574 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,575 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:40,575 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:40,575 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:40,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:40,575 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,576 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:40,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:40,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:40,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:40,576 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:40,576 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:40,576 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:40,576 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:40,576 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:40,577 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:40,581 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:40,581 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:40,581 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:40,581 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:40,581 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:40,581 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:40,581 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:40,581 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:40,581 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,581 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:40,581 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:40,582 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,582 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:40,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:40,582 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,582 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:40,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:40,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:40,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:40,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:40,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:40,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f52871a[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:40,584 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:40,584 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,585 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,585 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:40,585 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:40,585 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:40,585 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,585 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:40,586 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,586 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:40,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:40,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:40,586 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:40,586 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:40,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:40,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:40,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:40,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:40,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:40,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:40,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:40,588 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:40,588 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:40,588 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:40,588 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:40,588 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:40,589 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,589 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,589 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:40,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:40,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:40,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:40,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:40,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:40,590 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:40,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:40,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:40,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,591 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:40,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:40,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,593 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,593 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:40,593 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:40,593 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:40,593 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:40,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:40,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:40,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:40,593 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:40,593 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:40,593 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,593 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:40,593 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,594 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:40,594 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:40,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:40,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:40,641 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/5c0b89ae58d3487885379ec9e25ef0a6 2023-04-28 14:57:40,647 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/e4ca72a0b0d14de68efbad79d12f4d45 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/e4ca72a0b0d14de68efbad79d12f4d45 2023-04-28 14:57:40,652 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/e4ca72a0b0d14de68efbad79d12f4d45, entries=200, sequenceid=425, filesize=32.3 K 2023-04-28 14:57:40,653 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/1fbe0a5fed2b4e6b9b6a50a3f39daac7 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/1fbe0a5fed2b4e6b9b6a50a3f39daac7 2023-04-28 14:57:40,658 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/1fbe0a5fed2b4e6b9b6a50a3f39daac7, entries=150, sequenceid=425, filesize=11.9 K 2023-04-28 14:57:40,660 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/5c0b89ae58d3487885379ec9e25ef0a6 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/5c0b89ae58d3487885379ec9e25ef0a6 2023-04-28 14:57:40,667 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/5c0b89ae58d3487885379ec9e25ef0a6, entries=150, sequenceid=425, filesize=11.9 K 2023-04-28 14:57:40,668 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~167.72 KB/171750, heapSize ~461.58 KB/472656, currentSize=20.13 KB/20610 for 3df9cf7b9785edd3a0db773c7f6b886b in 1276ms, sequenceid=425, compaction requested=true 2023-04-28 14:57:40,668 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:40,668 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:57:40,668 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:40,668 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:57:40,668 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:57:40,668 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:57:40,670 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 118670 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:57:40,670 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46710 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:57:40,670 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): 3df9cf7b9785edd3a0db773c7f6b886b/A is initiating minor compaction (all files) 2023-04-28 14:57:40,670 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 3df9cf7b9785edd3a0db773c7f6b886b/B is initiating minor compaction (all files) 2023-04-28 14:57:40,670 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3df9cf7b9785edd3a0db773c7f6b886b/A in TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:40,670 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3df9cf7b9785edd3a0db773c7f6b886b/B in TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:40,670 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/1b6adf53932a4740b76bef50ae4a4381, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/3875048f3b594425bef5faaeffd7a55a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/f367eb15728c48de9ff12866c265b5c7, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/1fbe0a5fed2b4e6b9b6a50a3f39daac7] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp, totalSize=45.6 K 2023-04-28 14:57:40,670 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/5cfe200f93ec46cebe08ba5fb7b4e7e1, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/bcc4599df23b4ecf892ae44c2be548cd, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/e6d10f7ecb2e48458de94e5867c5771c, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/e4ca72a0b0d14de68efbad79d12f4d45] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp, totalSize=115.9 K 2023-04-28 14:57:40,671 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 1b6adf53932a4740b76bef50ae4a4381, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1682693855687 2023-04-28 14:57:40,671 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 5cfe200f93ec46cebe08ba5fb7b4e7e1, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1682693855687 2023-04-28 14:57:40,671 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 3875048f3b594425bef5faaeffd7a55a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1682693856307 2023-04-28 14:57:40,671 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting bcc4599df23b4ecf892ae44c2be548cd, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1682693856307 2023-04-28 14:57:40,671 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting f367eb15728c48de9ff12866c265b5c7, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1682693858440 2023-04-28 14:57:40,671 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting e6d10f7ecb2e48458de94e5867c5771c, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1682693858440 2023-04-28 14:57:40,672 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 1fbe0a5fed2b4e6b9b6a50a3f39daac7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1682693858751 2023-04-28 14:57:40,672 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting e4ca72a0b0d14de68efbad79d12f4d45, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1682693858751 2023-04-28 14:57:40,687 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 3df9cf7b9785edd3a0db773c7f6b886b#B#compaction#154 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:40,694 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:40,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:40,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:40,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:40,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:40,699 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:40,699 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:40,699 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:40,699 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:40,700 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,700 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3df9cf7b9785edd3a0db773c7f6b886b#A#compaction#153 average throughput is 1.10 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:40,700 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:40,700 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:40,700 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:40,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:40,701 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,701 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:40,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:40,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:40,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:40,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:40,702 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:40,710 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:40,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:40,710 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:40,710 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:40,710 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:40,720 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:40,720 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:40,720 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:40,720 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:40,720 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3df9cf7b9785edd3a0db773c7f6b886b 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-28 14:57:40,721 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=A 2023-04-28 14:57:40,721 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:40,721 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=B 2023-04-28 14:57:40,721 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:40,721 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=C 2023-04-28 14:57:40,721 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:40,734 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042822c5076e0c39483aaba81433cb9ca129 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042822c5076e0c39483aaba81433cb9ca129 2023-04-28 14:57:40,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:40,806 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:40,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682693860814, max timestamp=1682693860819Num uniques 859; 2023-04-28 14:57:40,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 14:57:40,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682693860814, max timestamp=1682693860819Num uniques 859; 2023-04-28 14:57:40,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 14:57:40,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682693860814, max timestamp=1682693860819Num uniques 859; 2023-04-28 14:57:40,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 14:57:40,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682693860847, max timestamp=1682693860856Num uniques 1118; 2023-04-28 14:57:40,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-28 14:57:40,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682693860847, max timestamp=1682693860856Num uniques 1118; 2023-04-28 14:57:40,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=8102, suffix heap size=144240,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=177 2023-04-28 14:57:40,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682693860847, max timestamp=1682693860856Num uniques 1118; 2023-04-28 14:57:40,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-28 14:57:40,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=427, cellSize=19552, totalHeapSize=44648, min timestamp=1682693860870, max timestamp=1682693860880Num uniques 1077; 2023-04-28 14:57:40,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=427, cellSize=19552, totalHeapSize=44648, min timestamp=1682693860870, max timestamp=1682693860880Num uniques 1036; 2023-04-28 14:57:40,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=19552, suffix heap size=139976,new segment heap size=44648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=427 2023-04-28 14:57:40,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=19552, suffix heap size=135712,new segment heap size=44648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=427 2023-04-28 14:57:40,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=427, cellSize=19552, totalHeapSize=44648, min timestamp=1682693860870, max timestamp=1682693860880Num uniques 1077; 2023-04-28 14:57:40,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=19552, suffix heap size=139976,new segment heap size=44648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=427 2023-04-28 14:57:40,899 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:40,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48944 deadline: 1682693920897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682693860894, max timestamp=1682693860898Num uniques 1286; 2023-04-28 14:57:40,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=58896, new segment data size=13096, suffix heap size=161712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1286, new segment cells count=286 2023-04-28 14:57:40,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682693860894, max timestamp=1682693860898Num uniques 1286; 2023-04-28 14:57:40,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=58896, new segment data size=13096, suffix heap size=161712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1286, new segment cells count=286 2023-04-28 14:57:40,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682693860894, max timestamp=1682693860898Num uniques 1286; 2023-04-28 14:57:40,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=58896, new segment data size=13096, suffix heap size=161712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1286, new segment cells count=286 2023-04-28 14:57:40,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1682693860939, max timestamp=1682693860947Num uniques 1145; 2023-04-28 14:57:40,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=15800, suffix heap size=147048,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=345 2023-04-28 14:57:40,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1682693860939, max timestamp=1682693860947Num uniques 1145; 2023-04-28 14:57:40,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1682693860939, max timestamp=1682693860947Num uniques 1145; 2023-04-28 14:57:40,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=15800, suffix heap size=147048,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=345 2023-04-28 14:57:40,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=15800, suffix heap size=147048,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=345 2023-04-28 14:57:40,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:40,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48932 deadline: 1682693920968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:40,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682693860964, max timestamp=1682693860971Num uniques 1204; 2023-04-28 14:57:40,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=9340, suffix heap size=153184,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=204 2023-04-28 14:57:40,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682693860964, max timestamp=1682693860971Num uniques 1204; 2023-04-28 14:57:40,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=9340, suffix heap size=153184,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=204 2023-04-28 14:57:40,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682693860964, max timestamp=1682693860971Num uniques 1204; 2023-04-28 14:57:40,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=9340, suffix heap size=153184,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=204 2023-04-28 14:57:41,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1682693860971, max timestamp=1682693860999Num uniques 1063; 2023-04-28 14:57:41,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=14328, suffix heap size=138520,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=313 2023-04-28 14:57:41,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1682693860971, max timestamp=1682693860999Num uniques 1063; 2023-04-28 14:57:41,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1682693860971, max timestamp=1682693860999Num uniques 1063; 2023-04-28 14:57:41,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=14328, suffix heap size=138520,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=313 2023-04-28 14:57:41,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=14328, suffix heap size=138520,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=313 2023-04-28 14:57:41,023 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=372, cellSize=17032, totalHeapSize=38928, min timestamp=1682693861009, max timestamp=1682693861020Num uniques 1172; 2023-04-28 14:57:41,023 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=17032, suffix heap size=149856,new segment heap size=38928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=372 2023-04-28 14:57:41,023 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=372, cellSize=17032, totalHeapSize=38928, min timestamp=1682693861009, max timestamp=1682693861020Num uniques 1172; 2023-04-28 14:57:41,023 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=372, cellSize=17032, totalHeapSize=38928, min timestamp=1682693861009, max timestamp=1682693861020Num uniques 1172; 2023-04-28 14:57:41,023 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=17032, suffix heap size=149856,new segment heap size=38928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=372 2023-04-28 14:57:41,023 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=17032, suffix heap size=149856,new segment heap size=38928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=372 2023-04-28 14:57:41,042 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1682693861027, max timestamp=1682693861040Num uniques 1231; 2023-04-28 14:57:41,042 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=56376, new segment data size=12866, suffix heap size=155992,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1231, new segment cells count=281 2023-04-28 14:57:41,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1682693861027, max timestamp=1682693861040Num uniques 1231; 2023-04-28 14:57:41,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=56376, new segment data size=12866, suffix heap size=155992,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1231, new segment cells count=281 2023-04-28 14:57:41,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1682693861027, max timestamp=1682693861040Num uniques 1231; 2023-04-28 14:57:41,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=56376, new segment data size=12866, suffix heap size=155992,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1231, new segment cells count=281 2023-04-28 14:57:41,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1682693861049, max timestamp=1682693861061Num uniques 1140; 2023-04-28 14:57:41,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=15570, suffix heap size=146528,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=340 2023-04-28 14:57:41,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1682693861049, max timestamp=1682693861061Num uniques 1140; 2023-04-28 14:57:41,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1682693861049, max timestamp=1682693861061Num uniques 1140; 2023-04-28 14:57:41,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=15570, suffix heap size=146528,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=340 2023-04-28 14:57:41,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=15570, suffix heap size=146528,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=340 2023-04-28 14:57:41,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=549, cellSize=25144, totalHeapSize=57336, min timestamp=1682693861066, max timestamp=1682693861084Num uniques 1199; 2023-04-28 14:57:41,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=25144, suffix heap size=152664,new segment heap size=57336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=549 2023-04-28 14:57:41,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=549, cellSize=25144, totalHeapSize=57336, min timestamp=1682693861066, max timestamp=1682693861084Num uniques 1199; 2023-04-28 14:57:41,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=25144, suffix heap size=152664,new segment heap size=57336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=549 2023-04-28 14:57:41,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=549, cellSize=25144, totalHeapSize=57336, min timestamp=1682693861066, max timestamp=1682693861084Num uniques 1199; 2023-04-28 14:57:41,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=25144, suffix heap size=152664,new segment heap size=57336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=549 2023-04-28 14:57:41,125 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/c2b0c370b9404be6b506b3bbef78897c as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/c2b0c370b9404be6b506b3bbef78897c 2023-04-28 14:57:41,126 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:41,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48944 deadline: 1682693921121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:41,126 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:41,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693921122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:41,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:41,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48932 deadline: 1682693921126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:41,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:41,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48948 deadline: 1682693921126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:41,130 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3df9cf7b9785edd3a0db773c7f6b886b/B of 3df9cf7b9785edd3a0db773c7f6b886b into c2b0c370b9404be6b506b3bbef78897c(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:41,130 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:41,130 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b., storeName=3df9cf7b9785edd3a0db773c7f6b886b/B, priority=12, startTime=1682693860668; duration=0sec 2023-04-28 14:57:41,130 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:57:41,130 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:57:41,134 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34558 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:57:41,134 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 3df9cf7b9785edd3a0db773c7f6b886b/C is initiating minor compaction (all files) 2023-04-28 14:57:41,134 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3df9cf7b9785edd3a0db773c7f6b886b/C in TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:41,134 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/7e0a62e3a22b4a2e81238a9683b94f34, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/313facf96c8c4f62891cf6eafe5720af, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/5c0b89ae58d3487885379ec9e25ef0a6] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp, totalSize=33.7 K 2023-04-28 14:57:41,135 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 7e0a62e3a22b4a2e81238a9683b94f34, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1682693856307 2023-04-28 14:57:41,135 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 313facf96c8c4f62891cf6eafe5720af, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1682693858440 2023-04-28 14:57:41,136 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 5c0b89ae58d3487885379ec9e25ef0a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1682693858751 2023-04-28 14:57:41,139 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=431, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/22275da0e5474895be7676cb7c4adf27 2023-04-28 14:57:41,143 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 3df9cf7b9785edd3a0db773c7f6b886b#C#compaction#156 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:41,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:41,228 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:41,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48944 deadline: 1682693921227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:41,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693921227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:41,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:41,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48932 deadline: 1682693921230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:41,231 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:41,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48948 deadline: 1682693921230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:41,431 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:41,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48944 deadline: 1682693921429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:41,432 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:41,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693921429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:41,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:41,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48932 deadline: 1682693921433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:41,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:41,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48948 deadline: 1682693921433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:41,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:41,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48998 deadline: 1682693921518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:41,545 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/bed29c3a60824179909e56abc7d41d0b as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/bed29c3a60824179909e56abc7d41d0b 2023-04-28 14:57:41,549 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3df9cf7b9785edd3a0db773c7f6b886b/A of 3df9cf7b9785edd3a0db773c7f6b886b into bed29c3a60824179909e56abc7d41d0b(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:41,549 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:41,549 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b., storeName=3df9cf7b9785edd3a0db773c7f6b886b/A, priority=12, startTime=1682693860668; duration=0sec 2023-04-28 14:57:41,549 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:41,564 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/a4881e104f2c4dea99da8d67d74fb622 2023-04-28 14:57:41,568 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/6439e9703703425d87b7e10a4632a71f as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/6439e9703703425d87b7e10a4632a71f 2023-04-28 14:57:41,580 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3df9cf7b9785edd3a0db773c7f6b886b/C of 3df9cf7b9785edd3a0db773c7f6b886b into 6439e9703703425d87b7e10a4632a71f(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:41,580 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:41,580 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b., storeName=3df9cf7b9785edd3a0db773c7f6b886b/C, priority=13, startTime=1682693860668; duration=0sec 2023-04-28 14:57:41,580 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:41,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:41,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48944 deadline: 1682693921733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:41,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:41,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693921733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:41,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:41,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48948 deadline: 1682693921737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:41,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:41,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48932 deadline: 1682693921738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:41,987 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/ad9621b146124086b33e4b8b593d8354 2023-04-28 14:57:41,991 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/22275da0e5474895be7676cb7c4adf27 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/22275da0e5474895be7676cb7c4adf27 2023-04-28 14:57:41,995 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/22275da0e5474895be7676cb7c4adf27, entries=150, sequenceid=431, filesize=25.4 K 2023-04-28 14:57:41,996 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/a4881e104f2c4dea99da8d67d74fb622 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/a4881e104f2c4dea99da8d67d74fb622 2023-04-28 14:57:41,999 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/a4881e104f2c4dea99da8d67d74fb622, entries=150, sequenceid=431, filesize=11.9 K 2023-04-28 14:57:42,000 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/ad9621b146124086b33e4b8b593d8354 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/ad9621b146124086b33e4b8b593d8354 2023-04-28 14:57:42,004 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/ad9621b146124086b33e4b8b593d8354, entries=150, sequenceid=431, filesize=11.9 K 2023-04-28 14:57:42,004 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=174.43 KB/178620 for 3df9cf7b9785edd3a0db773c7f6b886b in 1284ms, sequenceid=431, compaction requested=false 2023-04-28 14:57:42,004 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:42,005 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:42,005 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:42,005 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:42,005 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,005 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:42,005 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:42,006 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,006 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:42,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:42,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:42,006 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:42,006 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:42,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:42,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:42,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:42,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:42,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:42,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a1bf0f1[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:42,007 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:42,007 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:42,008 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:42,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:42,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:42,008 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:42,008 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:42,008 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:42,008 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,008 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:42,009 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:42,009 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,009 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:42,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:42,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:42,009 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:42,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:42,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:42,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:42,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:42,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:42,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:42,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:42,011 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:42,011 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:42,011 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:42,011 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:42,011 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:42,012 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:42,012 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:42,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:42,012 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:42,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:42,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:42,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:42,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:42,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:42,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:42,012 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:42,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:42,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:42,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:42,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:42,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:42,018 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:42,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,031 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,031 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:42,031 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:42,031 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:42,031 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:42,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:42,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:42,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:42,031 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:42,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:42,031 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,031 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:42,031 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:42,031 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:42,031 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:42,031 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:42,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:42,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:42,131 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:42,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:42,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:42,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:42,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:42,134 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:42,134 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:42,134 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:42,134 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:42,135 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:42,135 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:42,135 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:42,135 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:42,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:42,136 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,136 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:42,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:42,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:42,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:42,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:42,136 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:42,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:42,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:42,136 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:42,137 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:42,137 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:42,137 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:42,137 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:42,137 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:42,137 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:42,137 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3df9cf7b9785edd3a0db773c7f6b886b 3/3 column families, dataSize=174.43 KB heapSize=467.95 KB 2023-04-28 14:57:42,137 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=A 2023-04-28 14:57:42,137 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:42,137 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=B 2023-04-28 14:57:42,137 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:42,137 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=C 2023-04-28 14:57:42,137 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:42,148 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042829b76dd665e34f198c974aef86cca314 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042829b76dd665e34f198c974aef86cca314 2023-04-28 14:57:42,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:42,239 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:42,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:42,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693922240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,242 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:42,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48932 deadline: 1682693922240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,243 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:42,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48944 deadline: 1682693922241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,246 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:42,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48948 deadline: 1682693922244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:42,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693922342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:42,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48944 deadline: 1682693922343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:42,548 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:42,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48944 deadline: 1682693922547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693922547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,555 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=641, memsize=58.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/762d43443d484ad78dde75b6835857dd 2023-04-28 14:57:42,592 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=641 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/987bd6956d704f5b913db17234102a3a 2023-04-28 14:57:42,606 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=641 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/b5302c79bbc64091a75a40728f29755e 2023-04-28 14:57:42,610 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/762d43443d484ad78dde75b6835857dd as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/762d43443d484ad78dde75b6835857dd 2023-04-28 14:57:42,614 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/762d43443d484ad78dde75b6835857dd, entries=150, sequenceid=641, filesize=25.4 K 2023-04-28 14:57:42,615 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/987bd6956d704f5b913db17234102a3a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/987bd6956d704f5b913db17234102a3a 2023-04-28 14:57:42,624 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/987bd6956d704f5b913db17234102a3a, entries=150, sequenceid=641, filesize=11.9 K 2023-04-28 14:57:42,625 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/b5302c79bbc64091a75a40728f29755e as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/b5302c79bbc64091a75a40728f29755e 2023-04-28 14:57:42,630 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/b5302c79bbc64091a75a40728f29755e, entries=150, sequenceid=641, filesize=11.9 K 2023-04-28 14:57:42,630 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~174.43 KB/178620, heapSize ~467.91 KB/479136, currentSize=20.13 KB/20610 for 3df9cf7b9785edd3a0db773c7f6b886b in 493ms, sequenceid=641, compaction requested=true 2023-04-28 14:57:42,631 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:42,631 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:42,631 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:42,631 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:42,631 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,636 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:42,636 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:42,637 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:42,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:42,638 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:42,638 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:42,638 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:42,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:42,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:42,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:42,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:42,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:42,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f256e0[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:42,641 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:42,641 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:42,641 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:42,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:42,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:42,642 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:42,642 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:42,642 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:42,642 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,642 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:42,643 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:42,643 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,643 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:42,643 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:42,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:42,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:42,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:42,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:42,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:42,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:42,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:42,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:42,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:42,645 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:42,645 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:42,645 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:42,645 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:42,645 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:42,646 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:42,646 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:42,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:42,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:42,646 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:42,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:42,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:42,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:42,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:42,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:42,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:42,647 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:42,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:42,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:42,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:42,650 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:42,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:42,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,652 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:42,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:42,652 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:42,652 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:42,652 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:42,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:42,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:42,652 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:42,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:42,652 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:42,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:42,652 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:42,652 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:42,652 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:42,652 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:42,652 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:42,652 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:42,753 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:42,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:42,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:42,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:42,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:42,756 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:42,756 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:42,756 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:42,756 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:42,757 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:42,757 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:42,757 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:42,757 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:42,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:42,758 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:42,758 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:42,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:42,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:42,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:42,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:42,759 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:42,759 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:42,759 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:42,759 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:42,759 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:42,759 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:42,770 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:42,770 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:42,774 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:42,774 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:42,774 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3df9cf7b9785edd3a0db773c7f6b886b 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-28 14:57:42,774 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=A 2023-04-28 14:57:42,774 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:42,775 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=B 2023-04-28 14:57:42,775 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:42,775 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=C 2023-04-28 14:57:42,775 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:42,815 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428d167123328c346ef92b54463c86bf538 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428d167123328c346ef92b54463c86bf538 2023-04-28 14:57:42,824 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=647, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/f20dd2510037468fa31ab221288a5f6e 2023-04-28 14:57:42,864 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=647 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/85ac893d82634fc0b7466c337c18f57f 2023-04-28 14:57:42,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:42,872 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:42,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682693862920, max timestamp=1682693862932Num uniques 859; 2023-04-28 14:57:42,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682693862920, max timestamp=1682693862932Num uniques 859; 2023-04-28 14:57:42,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 14:57:42,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682693862920, max timestamp=1682693862932Num uniques 859; 2023-04-28 14:57:42,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 14:57:42,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 14:57:42,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682693862956, max timestamp=1682693862979Num uniques 1118; 2023-04-28 14:57:42,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-28 14:57:42,983 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682693862956, max timestamp=1682693862979Num uniques 1118; 2023-04-28 14:57:42,983 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-28 14:57:42,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682693862956, max timestamp=1682693862979Num uniques 1118; 2023-04-28 14:57:42,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-28 14:57:43,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682693863008, max timestamp=1682693863020Num uniques 1077; 2023-04-28 14:57:43,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-28 14:57:43,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682693863008, max timestamp=1682693863020Num uniques 1077; 2023-04-28 14:57:43,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-28 14:57:43,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682693863008, max timestamp=1682693863020Num uniques 1077; 2023-04-28 14:57:43,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-28 14:57:43,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682693863055, max timestamp=1682693863060Num uniques 1086; 2023-04-28 14:57:43,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-28 14:57:43,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682693863055, max timestamp=1682693863060Num uniques 1086; 2023-04-28 14:57:43,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-28 14:57:43,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682693863055, max timestamp=1682693863060Num uniques 1086; 2023-04-28 14:57:43,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-28 14:57:43,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1682693863071, max timestamp=1682693863092Num uniques 1095; 2023-04-28 14:57:43,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=18090, suffix heap size=141848,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=395 2023-04-28 14:57:43,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1682693863071, max timestamp=1682693863092Num uniques 1095; 2023-04-28 14:57:43,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=18090, suffix heap size=141848,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=395 2023-04-28 14:57:43,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1682693863071, max timestamp=1682693863092Num uniques 1095; 2023-04-28 14:57:43,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=18090, suffix heap size=141848,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=395 2023-04-28 14:57:43,128 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:43,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48944 deadline: 1682693923125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:43,128 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:43,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693923125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:43,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:43,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693923229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:43,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:43,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48944 deadline: 1682693923229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:43,253 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:43,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48932 deadline: 1682693923249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:43,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:43,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48948 deadline: 1682693923251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:43,280 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=647 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/6e58607178014eec9f4f83362b2182e5 2023-04-28 14:57:43,284 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/f20dd2510037468fa31ab221288a5f6e as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/f20dd2510037468fa31ab221288a5f6e 2023-04-28 14:57:43,288 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/f20dd2510037468fa31ab221288a5f6e, entries=50, sequenceid=647, filesize=11.7 K 2023-04-28 14:57:43,288 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/85ac893d82634fc0b7466c337c18f57f as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/85ac893d82634fc0b7466c337c18f57f 2023-04-28 14:57:43,300 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/85ac893d82634fc0b7466c337c18f57f, entries=50, sequenceid=647, filesize=7.1 K 2023-04-28 14:57:43,306 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/6e58607178014eec9f4f83362b2182e5 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/6e58607178014eec9f4f83362b2182e5 2023-04-28 14:57:43,311 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/6e58607178014eec9f4f83362b2182e5, entries=50, sequenceid=647, filesize=7.1 K 2023-04-28 14:57:43,313 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=167.72 KB/171750 for 3df9cf7b9785edd3a0db773c7f6b886b in 539ms, sequenceid=647, compaction requested=true 2023-04-28 14:57:43,313 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:43,313 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:43,313 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:43,313 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:43,313 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:43,313 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:43,313 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:43,315 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:43,315 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:43,315 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:43,315 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:43,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:43,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:43,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:43,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:43,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:43,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:43,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:43,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:43,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:43,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@391a1574[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:43,317 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:43,317 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:43,318 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:43,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:43,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:43,318 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:43,318 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:43,318 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:43,318 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:43,318 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:43,319 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:43,319 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:43,319 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:43,319 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:43,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:43,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:43,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:43,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:43,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:43,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:43,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:43,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:43,322 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:43,322 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:43,322 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:43,322 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:43,322 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:43,323 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:43,323 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:43,323 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:43,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:43,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:43,323 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:43,324 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:43,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:43,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:43,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:43,325 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:43,326 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:43,326 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:43,326 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:43,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:43,326 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:43,326 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:43,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:43,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:43,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:43,326 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:43,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:43,326 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:43,326 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:43,326 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:43,326 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:43,326 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:43,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:43,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:43,426 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:43,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:43,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:43,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:43,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:43,429 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:43,429 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:43,429 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:43,429 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:43,429 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:43,429 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:43,430 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:43,430 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:43,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:43,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:43,431 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:43,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:43,431 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:43,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:43,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:43,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:43,432 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:43,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:43,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:43,432 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:43,432 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:43,432 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:43,432 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:43,432 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:43,432 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:43,432 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:43,432 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3df9cf7b9785edd3a0db773c7f6b886b 3/3 column families, dataSize=167.72 KB heapSize=462.47 KB 2023-04-28 14:57:43,432 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=A 2023-04-28 14:57:43,433 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:43,433 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=B 2023-04-28 14:57:43,433 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:43,433 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=C 2023-04-28 14:57:43,433 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:57:43,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:43,438 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:43,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:43,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693923444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:43,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:43,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48944 deadline: 1682693923444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:43,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:43,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48998 deadline: 1682693923526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:43,529 DEBUG [Thread-808] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4128 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b., hostname=jenkins-hbase17.apache.org,37181,1682693812482, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 14:57:43,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:43,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693923550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:43,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:43,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48944 deadline: 1682693923551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:43,755 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:43,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693923754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:43,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:43,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48944 deadline: 1682693923754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:43,847 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428e656403c12d941b197ca7e64d9750dd6 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428e656403c12d941b197ca7e64d9750dd6 2023-04-28 14:57:44,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:44,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693924057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:44,062 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:44,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48944 deadline: 1682693924058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:44,253 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=753, memsize=55.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/ed96c426db754d50b38ded36ca23d0ae 2023-04-28 14:57:44,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:44,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693924559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:44,564 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:44,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48944 deadline: 1682693924564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:44,662 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=753 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/a292bfb95a2c4d00b5c5ad5f59701513 2023-04-28 14:57:45,072 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=753 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/d85ee140c39f44ce91ffeb9b5ad52654 2023-04-28 14:57:45,076 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/ed96c426db754d50b38ded36ca23d0ae as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/ed96c426db754d50b38ded36ca23d0ae 2023-04-28 14:57:45,079 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/ed96c426db754d50b38ded36ca23d0ae, entries=150, sequenceid=753, filesize=25.4 K 2023-04-28 14:57:45,080 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/a292bfb95a2c4d00b5c5ad5f59701513 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/a292bfb95a2c4d00b5c5ad5f59701513 2023-04-28 14:57:45,083 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/a292bfb95a2c4d00b5c5ad5f59701513, entries=150, sequenceid=753, filesize=11.9 K 2023-04-28 14:57:45,084 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/d85ee140c39f44ce91ffeb9b5ad52654 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/d85ee140c39f44ce91ffeb9b5ad52654 2023-04-28 14:57:45,088 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/d85ee140c39f44ce91ffeb9b5ad52654, entries=150, sequenceid=753, filesize=11.9 K 2023-04-28 14:57:45,089 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~167.72 KB/171750, heapSize ~462.42 KB/473520, currentSize=26.84 KB/27480 for 3df9cf7b9785edd3a0db773c7f6b886b in 1657ms, sequenceid=753, compaction requested=true 2023-04-28 14:57:45,089 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:45,089 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:45,089 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:45,089 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:45,089 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:45,089 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:45,089 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:45,090 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:45,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:45,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:45,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:45,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:45,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:45,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:45,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:45,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:45,093 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:45,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:45,098 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:45,098 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:45,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@723b3f1a[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:45,098 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:45,099 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:45,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:45,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:45,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:45,099 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:45,100 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:45,100 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:45,100 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:45,100 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:45,100 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:45,100 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:45,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:45,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:45,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:45,101 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:45,101 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:45,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:45,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:45,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:45,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:45,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:45,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:45,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:45,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:45,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:45,103 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:45,103 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:45,103 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:45,103 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:45,103 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:45,104 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:45,104 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:45,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:45,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:45,104 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:45,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:45,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:45,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:45,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:45,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:45,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:45,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:45,105 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:45,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:45,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:45,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:45,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:45,106 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:45,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:45,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:45,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:45,109 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:45,109 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:45,109 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:45,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:45,109 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:45,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:45,109 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:45,109 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:45,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:57:45,109 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:45,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:45,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:45,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:45,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:45,109 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:45,109 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:45,109 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:45,109 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:45,109 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:45,210 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:45,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:45,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:57:45,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:57:45,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:57:45,221 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:57:45,221 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:45,221 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:57:45,221 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:57:45,222 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:45,222 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:45,224 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:45,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:45,224 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:45,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:45,224 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:45,224 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:57:45,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:45,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:45,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:57:45,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:45,225 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:57:45,225 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:57:45,225 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:57:45,225 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:57:45,225 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:57:45,226 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:57:45,226 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:57:45,226 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:57:45,226 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:45,226 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. started... 2023-04-28 14:57:45,226 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3df9cf7b9785edd3a0db773c7f6b886b 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-28 14:57:45,226 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=A 2023-04-28 14:57:45,226 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:45,226 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=B 2023-04-28 14:57:45,227 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:45,227 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=C 2023-04-28 14:57:45,227 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:45,241 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428ff1bd1fa453b4eb082b51c57bf64075e to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ff1bd1fa453b4eb082b51c57bf64075e 2023-04-28 14:57:45,252 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=760, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/b1508732d94c4a89ab9dfc4087a1fc67 2023-04-28 14:57:45,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:45,274 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. as already flushing 2023-04-28 14:57:45,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682693865297, max timestamp=1682693865305Num uniques 859; 2023-04-28 14:57:45,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682693865297, max timestamp=1682693865305Num uniques 859; 2023-04-28 14:57:45,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:57:45,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:57:45,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682693865297, max timestamp=1682693865305Num uniques 859; 2023-04-28 14:57:45,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:57:45,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1682693865344, max timestamp=1682693865362Num uniques 1068; 2023-04-28 14:57:45,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1682693865344, max timestamp=1682693865362Num uniques 1068; 2023-04-28 14:57:45,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1682693865344, max timestamp=1682693865362Num uniques 1068; 2023-04-28 14:57:45,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=14558, suffix heap size=139040,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=318 2023-04-28 14:57:45,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=14558, suffix heap size=139040,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=318 2023-04-28 14:57:45,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=14558, suffix heap size=139040,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=318 2023-04-28 14:57:45,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:45,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48932 deadline: 1682693925395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:45,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682693865386, max timestamp=1682693865398Num uniques 1177; 2023-04-28 14:57:45,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682693865386, max timestamp=1682693865398Num uniques 1177; 2023-04-28 14:57:45,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=8102, suffix heap size=150376,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=177 2023-04-28 14:57:45,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682693865386, max timestamp=1682693865398Num uniques 1177; 2023-04-28 14:57:45,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=8102, suffix heap size=150376,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=177 2023-04-28 14:57:45,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=8102, suffix heap size=150376,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=177 2023-04-28 14:57:45,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682693865437, max timestamp=1682693865456Num uniques 1036; 2023-04-28 14:57:45,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-28 14:57:45,459 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682693865437, max timestamp=1682693865456Num uniques 1036; 2023-04-28 14:57:45,459 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-28 14:57:45,459 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682693865437, max timestamp=1682693865456Num uniques 1036; 2023-04-28 14:57:45,459 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-28 14:57:45,502 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682693865489, max timestamp=1682693865500Num uniques 1095; 2023-04-28 14:57:45,502 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-28 14:57:45,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682693865489, max timestamp=1682693865500Num uniques 1095; 2023-04-28 14:57:45,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-28 14:57:45,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682693865489, max timestamp=1682693865500Num uniques 1095; 2023-04-28 14:57:45,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-28 14:57:45,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682693865517, max timestamp=1682693865524Num uniques 1104; 2023-04-28 14:57:45,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-28 14:57:45,526 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:45,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48948 deadline: 1682693925524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:45,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682693865517, max timestamp=1682693865524Num uniques 1104; 2023-04-28 14:57:45,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-28 14:57:45,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682693865517, max timestamp=1682693865524Num uniques 1104; 2023-04-28 14:57:45,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-28 14:57:45,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682693865559, max timestamp=1682693865573Num uniques 1063; 2023-04-28 14:57:45,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-28 14:57:45,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682693865559, max timestamp=1682693865573Num uniques 1063; 2023-04-28 14:57:45,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-28 14:57:45,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682693865559, max timestamp=1682693865573Num uniques 1063; 2023-04-28 14:57:45,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-28 14:57:45,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1682693865582, max timestamp=1682693865593Num uniques 1022; 2023-04-28 14:57:45,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=12452, suffix heap size=134256,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=272 2023-04-28 14:57:45,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1682693865582, max timestamp=1682693865593Num uniques 1022; 2023-04-28 14:57:45,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1682693865582, max timestamp=1682693865593Num uniques 1022; 2023-04-28 14:57:45,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=12452, suffix heap size=134256,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=272 2023-04-28 14:57:45,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=12452, suffix heap size=134256,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=272 2023-04-28 14:57:45,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:45,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48944 deadline: 1682693925613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:45,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:45,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 382 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693925614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:45,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682693865607, max timestamp=1682693865615Num uniques 1131; 2023-04-28 14:57:45,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=8286, suffix heap size=145592,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=181 2023-04-28 14:57:45,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682693865607, max timestamp=1682693865615Num uniques 1131; 2023-04-28 14:57:45,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=8286, suffix heap size=145592,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=181 2023-04-28 14:57:45,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682693865607, max timestamp=1682693865615Num uniques 1131; 2023-04-28 14:57:45,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=8286, suffix heap size=145592,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=181 2023-04-28 14:57:45,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1682693865640, max timestamp=1682693865653Num uniques 1040; 2023-04-28 14:57:45,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-04-28 14:57:45,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682693865644, max timestamp=1682693865653Num uniques 1040; 2023-04-28 14:57:45,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-28 14:57:45,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682693865644, max timestamp=1682693865653Num uniques 1040; 2023-04-28 14:57:45,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=6870, suffix heap size=136128,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=150 2023-04-28 14:57:45,667 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=760 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/3b7cada7d0ce45f4851bd4cd845d3cc7 2023-04-28 14:57:45,701 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=399, cellSize=18274, totalHeapSize=41736, min timestamp=1682693865655, max timestamp=1682693865700Num uniques 1009; 2023-04-28 14:57:45,701 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=18274, suffix heap size=132904,new segment heap size=41736 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=399 2023-04-28 14:57:45,702 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=399, cellSize=18274, totalHeapSize=41736, min timestamp=1682693865655, max timestamp=1682693865700Num uniques 1149; 2023-04-28 14:57:45,702 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=399, cellSize=18274, totalHeapSize=41736, min timestamp=1682693865655, max timestamp=1682693865700Num uniques 1049; 2023-04-28 14:57:45,702 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=18274, suffix heap size=147464,new segment heap size=41736 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=399 2023-04-28 14:57:45,702 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=18274, suffix heap size=137064,new segment heap size=41736 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=399 2023-04-28 14:57:45,725 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:45,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48932 deadline: 1682693925721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:45,725 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:45,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48948 deadline: 1682693925721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:45,725 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:45,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48944 deadline: 1682693925722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:45,725 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:45,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 386 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693925723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:45,825 DEBUG [Thread-817] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x17a1bd04 to 127.0.0.1:61762 2023-04-28 14:57:45,825 DEBUG [Thread-817] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:57:45,827 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:45,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48932 deadline: 1682693925826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:45,827 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:45,827 DEBUG [Thread-813] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52902392 to 127.0.0.1:61762 2023-04-28 14:57:45,827 DEBUG [Thread-813] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:57:45,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48948 deadline: 1682693925826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:45,827 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:45,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48944 deadline: 1682693925826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:45,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:45,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 388 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693925826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:45,828 DEBUG [Thread-811] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bd79dff to 127.0.0.1:61762 2023-04-28 14:57:45,828 DEBUG [Thread-811] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:57:45,834 DEBUG [Thread-815] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4e1da5ba to 127.0.0.1:61762 2023-04-28 14:57:45,834 DEBUG [Thread-815] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:57:46,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:46,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48932 deadline: 1682693926028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:46,030 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:46,030 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:46,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:46,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 390 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48996 deadline: 1682693926030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:46,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48948 deadline: 1682693926030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:46,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48944 deadline: 1682693926030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:46,080 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=760 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/5a0a43a319644bd7b4cbce792b41db3b 2023-04-28 14:57:46,084 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/b1508732d94c4a89ab9dfc4087a1fc67 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/b1508732d94c4a89ab9dfc4087a1fc67 2023-04-28 14:57:46,089 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/b1508732d94c4a89ab9dfc4087a1fc67, entries=100, sequenceid=760, filesize=18.6 K 2023-04-28 14:57:46,090 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/3b7cada7d0ce45f4851bd4cd845d3cc7 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/3b7cada7d0ce45f4851bd4cd845d3cc7 2023-04-28 14:57:46,095 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/3b7cada7d0ce45f4851bd4cd845d3cc7, entries=100, sequenceid=760, filesize=9.5 K 2023-04-28 14:57:46,096 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/5a0a43a319644bd7b4cbce792b41db3b as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/5a0a43a319644bd7b4cbce792b41db3b 2023-04-28 14:57:46,100 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/5a0a43a319644bd7b4cbce792b41db3b, entries=100, sequenceid=760, filesize=9.5 K 2023-04-28 14:57:46,101 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=161.02 KB/164880 for 3df9cf7b9785edd3a0db773c7f6b886b in 875ms, sequenceid=760, compaction requested=true 2023-04-28 14:57:46,101 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:46,101 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:46,101 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:57:46,101 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:57:46,101 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:46,101 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:57:46,101 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:57:46,102 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:46,102 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:46,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:46,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:46,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:46,102 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:46,103 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:57:46,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:46,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:46,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:46,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:46,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:46,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:57:46,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37e2c5d4[Count = 0] remaining members to acquire global barrier 2023-04-28 14:57:46,104 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:57:46,105 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:46,105 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:46,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:46,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:46,105 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:57:46,105 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:57:46,105 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:57:46,106 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:46,106 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:57:46,106 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:46,106 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:57:46,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:46,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:46,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:46,107 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:46,107 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:57:46,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:46,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:46,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:46,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:46,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:57:46,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:57:46,110 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:57:46,110 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:57:46,110 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:57:46,110 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:57:46,110 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:57:46,111 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:46,111 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:46,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:46,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:57:46,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:57:46,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:46,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:46,111 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:46,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:57:46,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:46,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:46,112 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:46,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:46,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:46,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:57:46,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:46,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:46,113 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:46,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:57:46,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:57:46,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:46,115 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:46,115 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:57:46,115 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:46,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:57:46,115 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:57:46,115 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:57:46,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:57:46,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:57:46,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:57:46,115 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:57:46,115 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:46,115 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:46,116 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:57:46,116 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:57:46,116 DEBUG [Thread-810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:57:46,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:57:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:57:46,116 DEBUG [Thread-810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:57:46,216 DEBUG [Thread-810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:57:46,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:57:46,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:46,332 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3df9cf7b9785edd3a0db773c7f6b886b 3/3 column families, dataSize=167.72 KB heapSize=462.09 KB 2023-04-28 14:57:46,332 DEBUG [Thread-802] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x595355a0 to 127.0.0.1:61762 2023-04-28 14:57:46,332 DEBUG [Thread-802] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:57:46,333 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=A 2023-04-28 14:57:46,333 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 14:57:46,333 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=B 2023-04-28 14:57:46,334 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 14:57:46,334 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=C 2023-04-28 14:57:46,334 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 14:57:46,334 DEBUG [Thread-804] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x38da9522 to 127.0.0.1:61762 2023-04-28 14:57:46,334 DEBUG [Thread-804] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:57:46,336 DEBUG [Thread-800] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x389356cd to 127.0.0.1:61762 2023-04-28 14:57:46,336 DEBUG [Thread-800] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:57:46,338 DEBUG [Thread-806] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a84dba6 to 127.0.0.1:61762 2023-04-28 14:57:46,338 DEBUG [Thread-806] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:57:46,749 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428cf2a6528766b4c799b4cd14eae1da092 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428cf2a6528766b4c799b4cd14eae1da092 2023-04-28 14:57:47,155 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=970, memsize=58.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/9963f3bce3324ad8b30660102ad503b6 2023-04-28 14:57:47,562 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:57:47,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48998 deadline: 1682693927562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:47,566 DEBUG [Thread-808] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8164 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b., hostname=jenkins-hbase17.apache.org,37181,1682693812482, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 14:57:47,576 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=970 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/c658912d7571485db10d408fa6aff9bf 2023-04-28 14:57:47,996 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=970 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/4ac162fba1684237a778940964973835 2023-04-28 14:57:48,006 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/9963f3bce3324ad8b30660102ad503b6 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/9963f3bce3324ad8b30660102ad503b6 2023-04-28 14:57:48,013 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/9963f3bce3324ad8b30660102ad503b6, entries=150, sequenceid=970, filesize=25.4 K 2023-04-28 14:57:48,014 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/c658912d7571485db10d408fa6aff9bf as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/c658912d7571485db10d408fa6aff9bf 2023-04-28 14:57:48,022 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/c658912d7571485db10d408fa6aff9bf, entries=150, sequenceid=970, filesize=11.9 K 2023-04-28 14:57:48,023 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/4ac162fba1684237a778940964973835 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/4ac162fba1684237a778940964973835 2023-04-28 14:57:48,030 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/4ac162fba1684237a778940964973835, entries=150, sequenceid=970, filesize=11.9 K 2023-04-28 14:57:48,031 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~174.43 KB/178620, heapSize ~482.67 KB/494256, currentSize=13.42 KB/13740 for 3df9cf7b9785edd3a0db773c7f6b886b in 1699ms, sequenceid=970, compaction requested=true 2023-04-28 14:57:48,031 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:48,031 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:48,031 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-28 14:57:48,031 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-28 14:57:48,031 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:48,032 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:57:48,034 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 78518 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-28 14:57:48,034 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 3df9cf7b9785edd3a0db773c7f6b886b/B is initiating minor compaction (all files) 2023-04-28 14:57:48,034 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 161730 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-28 14:57:48,034 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): 3df9cf7b9785edd3a0db773c7f6b886b/A is initiating minor compaction (all files) 2023-04-28 14:57:48,034 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3df9cf7b9785edd3a0db773c7f6b886b/B in TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:48,034 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3df9cf7b9785edd3a0db773c7f6b886b/A in TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:48,034 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/c2b0c370b9404be6b506b3bbef78897c, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/a4881e104f2c4dea99da8d67d74fb622, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/987bd6956d704f5b913db17234102a3a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/85ac893d82634fc0b7466c337c18f57f, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/a292bfb95a2c4d00b5c5ad5f59701513, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/3b7cada7d0ce45f4851bd4cd845d3cc7, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/c658912d7571485db10d408fa6aff9bf] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp, totalSize=76.7 K 2023-04-28 14:57:48,034 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/bed29c3a60824179909e56abc7d41d0b, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/22275da0e5474895be7676cb7c4adf27, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/762d43443d484ad78dde75b6835857dd, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/f20dd2510037468fa31ab221288a5f6e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/ed96c426db754d50b38ded36ca23d0ae, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/b1508732d94c4a89ab9dfc4087a1fc67, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/9963f3bce3324ad8b30660102ad503b6] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp, totalSize=157.9 K 2023-04-28 14:57:48,035 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting c2b0c370b9404be6b506b3bbef78897c, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1682693858751 2023-04-28 14:57:48,035 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting bed29c3a60824179909e56abc7d41d0b, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1682693858751 2023-04-28 14:57:48,035 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting a4881e104f2c4dea99da8d67d74fb622, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1682693859394 2023-04-28 14:57:48,035 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 22275da0e5474895be7676cb7c4adf27, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1682693859394 2023-04-28 14:57:48,036 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 987bd6956d704f5b913db17234102a3a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1682693861119 2023-04-28 14:57:48,036 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 762d43443d484ad78dde75b6835857dd, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1682693861119 2023-04-28 14:57:48,036 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 85ac893d82634fc0b7466c337c18f57f, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=647, earliestPutTs=1682693862240 2023-04-28 14:57:48,036 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting f20dd2510037468fa31ab221288a5f6e, keycount=50, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=647, earliestPutTs=1682693862240 2023-04-28 14:57:48,037 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting ed96c426db754d50b38ded36ca23d0ae, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=753, earliestPutTs=1682693863109 2023-04-28 14:57:48,037 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting a292bfb95a2c4d00b5c5ad5f59701513, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=753, earliestPutTs=1682693863109 2023-04-28 14:57:48,037 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 3b7cada7d0ce45f4851bd4cd845d3cc7, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=760, earliestPutTs=1682693863440 2023-04-28 14:57:48,038 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting b1508732d94c4a89ab9dfc4087a1fc67, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=760, earliestPutTs=1682693863440 2023-04-28 14:57:48,038 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting c658912d7571485db10d408fa6aff9bf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=970, earliestPutTs=1682693865721 2023-04-28 14:57:48,038 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 9963f3bce3324ad8b30660102ad503b6, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=970, earliestPutTs=1682693865721 2023-04-28 14:57:48,071 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 3df9cf7b9785edd3a0db773c7f6b886b#B#compaction#175 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:48,073 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3df9cf7b9785edd3a0db773c7f6b886b#A#compaction#174 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:48,494 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/0baf4d7fcb844919b44a7b596db306c5 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/0baf4d7fcb844919b44a7b596db306c5 2023-04-28 14:57:48,498 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 3df9cf7b9785edd3a0db773c7f6b886b/B of 3df9cf7b9785edd3a0db773c7f6b886b into 0baf4d7fcb844919b44a7b596db306c5(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:48,498 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:48,498 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b., storeName=3df9cf7b9785edd3a0db773c7f6b886b/B, priority=9, startTime=1682693868031; duration=0sec 2023-04-28 14:57:48,498 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:57:48,498 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-28 14:57:48,500 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 78518 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-28 14:57:48,500 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 3df9cf7b9785edd3a0db773c7f6b886b/C is initiating minor compaction (all files) 2023-04-28 14:57:48,500 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3df9cf7b9785edd3a0db773c7f6b886b/C in TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:48,500 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/6439e9703703425d87b7e10a4632a71f, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/ad9621b146124086b33e4b8b593d8354, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/b5302c79bbc64091a75a40728f29755e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/6e58607178014eec9f4f83362b2182e5, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/d85ee140c39f44ce91ffeb9b5ad52654, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/5a0a43a319644bd7b4cbce792b41db3b, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/4ac162fba1684237a778940964973835] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp, totalSize=76.7 K 2023-04-28 14:57:48,500 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 6439e9703703425d87b7e10a4632a71f, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1682693858751 2023-04-28 14:57:48,501 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting ad9621b146124086b33e4b8b593d8354, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1682693859394 2023-04-28 14:57:48,501 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting b5302c79bbc64091a75a40728f29755e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1682693861119 2023-04-28 14:57:48,501 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 6e58607178014eec9f4f83362b2182e5, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=647, earliestPutTs=1682693862240 2023-04-28 14:57:48,501 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting d85ee140c39f44ce91ffeb9b5ad52654, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=753, earliestPutTs=1682693863109 2023-04-28 14:57:48,502 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 5a0a43a319644bd7b4cbce792b41db3b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=760, earliestPutTs=1682693863440 2023-04-28 14:57:48,502 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 4ac162fba1684237a778940964973835, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=970, earliestPutTs=1682693865721 2023-04-28 14:57:48,512 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 3df9cf7b9785edd3a0db773c7f6b886b#C#compaction#176 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:57:48,922 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/db40a0fc6aa1437db02b6736cc8e5a2e as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/db40a0fc6aa1437db02b6736cc8e5a2e 2023-04-28 14:57:48,927 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 3df9cf7b9785edd3a0db773c7f6b886b/C of 3df9cf7b9785edd3a0db773c7f6b886b into db40a0fc6aa1437db02b6736cc8e5a2e(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:57:48,927 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:48,927 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b., storeName=3df9cf7b9785edd3a0db773c7f6b886b/C, priority=9, startTime=1682693868032; duration=0sec 2023-04-28 14:57:48,927 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:49,319 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/62d09bd3353945ef9ce07bc7b21e3c34 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/62d09bd3353945ef9ce07bc7b21e3c34 2023-04-28 14:57:49,323 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 3df9cf7b9785edd3a0db773c7f6b886b/A of 3df9cf7b9785edd3a0db773c7f6b886b into 62d09bd3353945ef9ce07bc7b21e3c34(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-28 14:57:49,323 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:49,323 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b., storeName=3df9cf7b9785edd3a0db773c7f6b886b/A, priority=9, startTime=1682693868031; duration=1sec 2023-04-28 14:57:49,323 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:57:57,645 DEBUG [Thread-808] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7c0b7748 to 127.0.0.1:61762 2023-04-28 14:57:57,645 DEBUG [Thread-808] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:57:57,645 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-28 14:57:57,646 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(394): wrote 228 2023-04-28 14:57:57,646 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(394): wrote 165 2023-04-28 14:57:57,646 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(394): wrote 166 2023-04-28 14:57:57,646 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(394): wrote 283 2023-04-28 14:57:57,646 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(394): wrote 54 2023-04-28 14:57:57,646 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-28 14:57:57,646 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(398): read 4251 2023-04-28 14:57:57,646 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(398): read 4252 2023-04-28 14:57:57,646 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-28 14:57:57,646 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(402): scanned 1791 2023-04-28 14:57:57,646 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(403): verified 5372 rows 2023-04-28 14:57:57,646 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(402): scanned 1803 2023-04-28 14:57:57,646 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(403): verified 5409 rows 2023-04-28 14:57:57,646 INFO [Listener at localhost.localdomain/40193] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-28 14:57:57,646 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x556231fa to 127.0.0.1:61762 2023-04-28 14:57:57,646 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:57:57,647 INFO [Listener at localhost.localdomain/40193] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-28 14:57:57,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.HMaster$11(2418): Client=jenkins//136.243.18.41 disable TestAcidGuarantees 2023-04-28 14:57:57,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-28 14:57:57,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-28 14:57:57,651 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682693877651"}]},"ts":"1682693877651"} 2023-04-28 14:57:57,652 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-28 14:57:57,653 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-28 14:57:57,653 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3df9cf7b9785edd3a0db773c7f6b886b, UNASSIGN}] 2023-04-28 14:57:57,654 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3df9cf7b9785edd3a0db773c7f6b886b, UNASSIGN 2023-04-28 14:57:57,655 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=3df9cf7b9785edd3a0db773c7f6b886b, regionState=CLOSING, regionLocation=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:57:57,655 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693877655"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682693877655"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682693877655"}]},"ts":"1682693877655"} 2023-04-28 14:57:57,656 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482}] 2023-04-28 14:57:57,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-28 14:57:57,806 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(109): Close 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:57,806 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1604): Closing 3df9cf7b9785edd3a0db773c7f6b886b, disabling compactions & flushes 2023-04-28 14:57:57,806 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:57,806 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:57,806 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. after waiting 0 ms 2023-04-28 14:57:57,806 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:57,807 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2745): Flushing 3df9cf7b9785edd3a0db773c7f6b886b 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-28 14:57:57,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=A 2023-04-28 14:57:57,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:57,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=B 2023-04-28 14:57:57,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:57,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3df9cf7b9785edd3a0db773c7f6b886b, store=C 2023-04-28 14:57:57,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:57:57,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-28 14:57:58,221 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042844e6aaec71cb456e946785bcaf6bb2be to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042844e6aaec71cb456e946785bcaf6bb2be 2023-04-28 14:57:58,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-28 14:57:58,626 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=979, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/0e2f1ff12720403db7f7c08be8f22bb8 2023-04-28 14:57:58,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-28 14:57:59,037 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=979 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/7f18b3df7ea5463aa34917f2de0d8e5a 2023-04-28 14:57:59,457 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=979 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/1f326cf1a5c64fc39e60102da65b4bf2 2023-04-28 14:57:59,467 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/A/0e2f1ff12720403db7f7c08be8f22bb8 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/0e2f1ff12720403db7f7c08be8f22bb8 2023-04-28 14:57:59,473 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/0e2f1ff12720403db7f7c08be8f22bb8, entries=100, sequenceid=979, filesize=18.6 K 2023-04-28 14:57:59,474 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/B/7f18b3df7ea5463aa34917f2de0d8e5a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/7f18b3df7ea5463aa34917f2de0d8e5a 2023-04-28 14:57:59,479 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/7f18b3df7ea5463aa34917f2de0d8e5a, entries=100, sequenceid=979, filesize=9.5 K 2023-04-28 14:57:59,480 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/.tmp/C/1f326cf1a5c64fc39e60102da65b4bf2 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/1f326cf1a5c64fc39e60102da65b4bf2 2023-04-28 14:57:59,484 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/1f326cf1a5c64fc39e60102da65b4bf2, entries=100, sequenceid=979, filesize=9.5 K 2023-04-28 14:57:59,485 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 3df9cf7b9785edd3a0db773c7f6b886b in 1679ms, sequenceid=979, compaction requested=false 2023-04-28 14:57:59,502 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/252e41c384574c9ab43ccd78f8197e8a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/eadb01fa3f6e46e4913b35c9fa56d2d2, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/5d573dc91ee74902b64ed84e4225ad37, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/dbed45591ee34d3a8b79f88e1269e12c, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/216f73997392447ea0d97ec77c777c63, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/15740794a58043d5bec0f7c3e8912888, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/d861af5d1fd043489ea9ae8b74363f33, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/2f1068b82a2440a6918bb6ae7297b602, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/25fc7e9715ad4320b85f7011c6e1b85d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/396612d922734853942f822e06397ed1, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/00d850e9e50c49cf8b0b7a1b524f8158, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/7fc7ff42602c42e5923e10489fa0db83, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/5cfe200f93ec46cebe08ba5fb7b4e7e1, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/bcc4599df23b4ecf892ae44c2be548cd, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/e6d10f7ecb2e48458de94e5867c5771c, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/e4ca72a0b0d14de68efbad79d12f4d45, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/bed29c3a60824179909e56abc7d41d0b, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/22275da0e5474895be7676cb7c4adf27, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/762d43443d484ad78dde75b6835857dd, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/f20dd2510037468fa31ab221288a5f6e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/ed96c426db754d50b38ded36ca23d0ae, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/b1508732d94c4a89ab9dfc4087a1fc67, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/9963f3bce3324ad8b30660102ad503b6] to archive 2023-04-28 14:57:59,504 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 14:57:59,505 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/252e41c384574c9ab43ccd78f8197e8a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/252e41c384574c9ab43ccd78f8197e8a 2023-04-28 14:57:59,506 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/eadb01fa3f6e46e4913b35c9fa56d2d2 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/eadb01fa3f6e46e4913b35c9fa56d2d2 2023-04-28 14:57:59,508 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/5d573dc91ee74902b64ed84e4225ad37 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/5d573dc91ee74902b64ed84e4225ad37 2023-04-28 14:57:59,509 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/dbed45591ee34d3a8b79f88e1269e12c to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/dbed45591ee34d3a8b79f88e1269e12c 2023-04-28 14:57:59,510 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/216f73997392447ea0d97ec77c777c63 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/216f73997392447ea0d97ec77c777c63 2023-04-28 14:57:59,511 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/15740794a58043d5bec0f7c3e8912888 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/15740794a58043d5bec0f7c3e8912888 2023-04-28 14:57:59,512 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/d861af5d1fd043489ea9ae8b74363f33 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/d861af5d1fd043489ea9ae8b74363f33 2023-04-28 14:57:59,513 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/2f1068b82a2440a6918bb6ae7297b602 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/2f1068b82a2440a6918bb6ae7297b602 2023-04-28 14:57:59,514 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/25fc7e9715ad4320b85f7011c6e1b85d to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/25fc7e9715ad4320b85f7011c6e1b85d 2023-04-28 14:57:59,516 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/396612d922734853942f822e06397ed1 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/396612d922734853942f822e06397ed1 2023-04-28 14:57:59,517 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/00d850e9e50c49cf8b0b7a1b524f8158 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/00d850e9e50c49cf8b0b7a1b524f8158 2023-04-28 14:57:59,518 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/7fc7ff42602c42e5923e10489fa0db83 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/7fc7ff42602c42e5923e10489fa0db83 2023-04-28 14:57:59,519 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/5cfe200f93ec46cebe08ba5fb7b4e7e1 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/5cfe200f93ec46cebe08ba5fb7b4e7e1 2023-04-28 14:57:59,521 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/bcc4599df23b4ecf892ae44c2be548cd to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/bcc4599df23b4ecf892ae44c2be548cd 2023-04-28 14:57:59,522 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/e6d10f7ecb2e48458de94e5867c5771c to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/e6d10f7ecb2e48458de94e5867c5771c 2023-04-28 14:57:59,523 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/e4ca72a0b0d14de68efbad79d12f4d45 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/e4ca72a0b0d14de68efbad79d12f4d45 2023-04-28 14:57:59,525 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/bed29c3a60824179909e56abc7d41d0b to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/bed29c3a60824179909e56abc7d41d0b 2023-04-28 14:57:59,526 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/22275da0e5474895be7676cb7c4adf27 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/22275da0e5474895be7676cb7c4adf27 2023-04-28 14:57:59,527 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/762d43443d484ad78dde75b6835857dd to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/762d43443d484ad78dde75b6835857dd 2023-04-28 14:57:59,528 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/f20dd2510037468fa31ab221288a5f6e to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/f20dd2510037468fa31ab221288a5f6e 2023-04-28 14:57:59,530 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/ed96c426db754d50b38ded36ca23d0ae to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/ed96c426db754d50b38ded36ca23d0ae 2023-04-28 14:57:59,531 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/b1508732d94c4a89ab9dfc4087a1fc67 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/b1508732d94c4a89ab9dfc4087a1fc67 2023-04-28 14:57:59,532 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/9963f3bce3324ad8b30660102ad503b6 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/9963f3bce3324ad8b30660102ad503b6 2023-04-28 14:57:59,534 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/2d4e4ea21f8547709e172516470d51e1, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/7e09a7156a494124bc73b5cefa1fd02b, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/1f6cf567d47546a7a39586cdf316e07a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/4e50eb4e744644678482602992129273, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/164049f739a0405bbde34cf81276a5f7, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/5849a96ff25d4af2a730788ff14bc7f9, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/12b9e9b6629c41d59efa5169e72ede33, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/0e47d7e42d1a4f0390880d8a800dcd16, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/454095b669934edeabfc182e83ee7601, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/a3095a4faa0c43d68cff36a4545ae1ad, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/f6c9de97a9504b49b28be51d5a67cd16, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/1b6adf53932a4740b76bef50ae4a4381, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/3ea8af070900425787b97f94726ce1a9, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/3875048f3b594425bef5faaeffd7a55a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/f367eb15728c48de9ff12866c265b5c7, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/c2b0c370b9404be6b506b3bbef78897c, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/1fbe0a5fed2b4e6b9b6a50a3f39daac7, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/a4881e104f2c4dea99da8d67d74fb622, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/987bd6956d704f5b913db17234102a3a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/85ac893d82634fc0b7466c337c18f57f, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/a292bfb95a2c4d00b5c5ad5f59701513, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/3b7cada7d0ce45f4851bd4cd845d3cc7, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/c658912d7571485db10d408fa6aff9bf] to archive 2023-04-28 14:57:59,535 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 14:57:59,536 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/2d4e4ea21f8547709e172516470d51e1 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/2d4e4ea21f8547709e172516470d51e1 2023-04-28 14:57:59,537 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/7e09a7156a494124bc73b5cefa1fd02b to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/7e09a7156a494124bc73b5cefa1fd02b 2023-04-28 14:57:59,538 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/1f6cf567d47546a7a39586cdf316e07a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/1f6cf567d47546a7a39586cdf316e07a 2023-04-28 14:57:59,539 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/4e50eb4e744644678482602992129273 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/4e50eb4e744644678482602992129273 2023-04-28 14:57:59,539 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/164049f739a0405bbde34cf81276a5f7 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/164049f739a0405bbde34cf81276a5f7 2023-04-28 14:57:59,540 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/5849a96ff25d4af2a730788ff14bc7f9 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/5849a96ff25d4af2a730788ff14bc7f9 2023-04-28 14:57:59,541 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/12b9e9b6629c41d59efa5169e72ede33 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/12b9e9b6629c41d59efa5169e72ede33 2023-04-28 14:57:59,542 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/0e47d7e42d1a4f0390880d8a800dcd16 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/0e47d7e42d1a4f0390880d8a800dcd16 2023-04-28 14:57:59,543 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/454095b669934edeabfc182e83ee7601 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/454095b669934edeabfc182e83ee7601 2023-04-28 14:57:59,544 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/a3095a4faa0c43d68cff36a4545ae1ad to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/a3095a4faa0c43d68cff36a4545ae1ad 2023-04-28 14:57:59,545 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/f6c9de97a9504b49b28be51d5a67cd16 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/f6c9de97a9504b49b28be51d5a67cd16 2023-04-28 14:57:59,546 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/1b6adf53932a4740b76bef50ae4a4381 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/1b6adf53932a4740b76bef50ae4a4381 2023-04-28 14:57:59,547 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/3ea8af070900425787b97f94726ce1a9 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/3ea8af070900425787b97f94726ce1a9 2023-04-28 14:57:59,548 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/3875048f3b594425bef5faaeffd7a55a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/3875048f3b594425bef5faaeffd7a55a 2023-04-28 14:57:59,549 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/f367eb15728c48de9ff12866c265b5c7 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/f367eb15728c48de9ff12866c265b5c7 2023-04-28 14:57:59,550 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/c2b0c370b9404be6b506b3bbef78897c to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/c2b0c370b9404be6b506b3bbef78897c 2023-04-28 14:57:59,551 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/1fbe0a5fed2b4e6b9b6a50a3f39daac7 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/1fbe0a5fed2b4e6b9b6a50a3f39daac7 2023-04-28 14:57:59,553 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/a4881e104f2c4dea99da8d67d74fb622 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/a4881e104f2c4dea99da8d67d74fb622 2023-04-28 14:57:59,554 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/987bd6956d704f5b913db17234102a3a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/987bd6956d704f5b913db17234102a3a 2023-04-28 14:57:59,556 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/85ac893d82634fc0b7466c337c18f57f to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/85ac893d82634fc0b7466c337c18f57f 2023-04-28 14:57:59,557 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/a292bfb95a2c4d00b5c5ad5f59701513 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/a292bfb95a2c4d00b5c5ad5f59701513 2023-04-28 14:57:59,559 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/3b7cada7d0ce45f4851bd4cd845d3cc7 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/3b7cada7d0ce45f4851bd4cd845d3cc7 2023-04-28 14:57:59,560 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/c658912d7571485db10d408fa6aff9bf to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/c658912d7571485db10d408fa6aff9bf 2023-04-28 14:57:59,563 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/96d4bea47b964c9985d562087b9e3fde, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/6ec64e2eb42047288c623f64ee1cebc1, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/f371330e759f4a3baa73ee8500a30e9a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/d830cceff4cb46358098ae7898363319, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/bb931a9b11de4518b41a948f65f8fb67, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/805edbc8e6a140c0a55503b0ab087c36, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/763afba87f634ba6ace64249776bb3e6, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/9a8f2e00703d4fd9b6a8c30d5ca4de8e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/46f9fd9f2c634804864c7de08e078af6, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/1c02482d3f744352b57022433de4edea, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/c6d383d37a294f4886e897fc6e79b68b, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/9d3fc29e05b547c7907ea5e8b457eee6, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/7e0a62e3a22b4a2e81238a9683b94f34, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/8ac89a44d78940758776143061eff010, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/313facf96c8c4f62891cf6eafe5720af, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/6439e9703703425d87b7e10a4632a71f, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/5c0b89ae58d3487885379ec9e25ef0a6, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/ad9621b146124086b33e4b8b593d8354, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/b5302c79bbc64091a75a40728f29755e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/6e58607178014eec9f4f83362b2182e5, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/d85ee140c39f44ce91ffeb9b5ad52654, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/5a0a43a319644bd7b4cbce792b41db3b, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/4ac162fba1684237a778940964973835] to archive 2023-04-28 14:57:59,563 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 14:57:59,565 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/96d4bea47b964c9985d562087b9e3fde to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/96d4bea47b964c9985d562087b9e3fde 2023-04-28 14:57:59,566 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/6ec64e2eb42047288c623f64ee1cebc1 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/6ec64e2eb42047288c623f64ee1cebc1 2023-04-28 14:57:59,567 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/f371330e759f4a3baa73ee8500a30e9a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/f371330e759f4a3baa73ee8500a30e9a 2023-04-28 14:57:59,568 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/d830cceff4cb46358098ae7898363319 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/d830cceff4cb46358098ae7898363319 2023-04-28 14:57:59,569 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/bb931a9b11de4518b41a948f65f8fb67 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/bb931a9b11de4518b41a948f65f8fb67 2023-04-28 14:57:59,570 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/805edbc8e6a140c0a55503b0ab087c36 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/805edbc8e6a140c0a55503b0ab087c36 2023-04-28 14:57:59,571 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/763afba87f634ba6ace64249776bb3e6 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/763afba87f634ba6ace64249776bb3e6 2023-04-28 14:57:59,572 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/9a8f2e00703d4fd9b6a8c30d5ca4de8e to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/9a8f2e00703d4fd9b6a8c30d5ca4de8e 2023-04-28 14:57:59,573 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/46f9fd9f2c634804864c7de08e078af6 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/46f9fd9f2c634804864c7de08e078af6 2023-04-28 14:57:59,574 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/1c02482d3f744352b57022433de4edea to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/1c02482d3f744352b57022433de4edea 2023-04-28 14:57:59,575 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/c6d383d37a294f4886e897fc6e79b68b to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/c6d383d37a294f4886e897fc6e79b68b 2023-04-28 14:57:59,576 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/9d3fc29e05b547c7907ea5e8b457eee6 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/9d3fc29e05b547c7907ea5e8b457eee6 2023-04-28 14:57:59,577 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/7e0a62e3a22b4a2e81238a9683b94f34 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/7e0a62e3a22b4a2e81238a9683b94f34 2023-04-28 14:57:59,578 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/8ac89a44d78940758776143061eff010 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/8ac89a44d78940758776143061eff010 2023-04-28 14:57:59,579 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/313facf96c8c4f62891cf6eafe5720af to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/313facf96c8c4f62891cf6eafe5720af 2023-04-28 14:57:59,580 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/6439e9703703425d87b7e10a4632a71f to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/6439e9703703425d87b7e10a4632a71f 2023-04-28 14:57:59,581 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/5c0b89ae58d3487885379ec9e25ef0a6 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/5c0b89ae58d3487885379ec9e25ef0a6 2023-04-28 14:57:59,582 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/ad9621b146124086b33e4b8b593d8354 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/ad9621b146124086b33e4b8b593d8354 2023-04-28 14:57:59,583 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/b5302c79bbc64091a75a40728f29755e to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/b5302c79bbc64091a75a40728f29755e 2023-04-28 14:57:59,584 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/6e58607178014eec9f4f83362b2182e5 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/6e58607178014eec9f4f83362b2182e5 2023-04-28 14:57:59,585 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/d85ee140c39f44ce91ffeb9b5ad52654 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/d85ee140c39f44ce91ffeb9b5ad52654 2023-04-28 14:57:59,586 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/5a0a43a319644bd7b4cbce792b41db3b to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/5a0a43a319644bd7b4cbce792b41db3b 2023-04-28 14:57:59,587 DEBUG [StoreCloser-TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/4ac162fba1684237a778940964973835 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/4ac162fba1684237a778940964973835 2023-04-28 14:57:59,590 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/recovered.edits/982.seqid, newMaxSeqId=982, maxSeqId=4 2023-04-28 14:57:59,592 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b. 2023-04-28 14:57:59,592 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1558): Region close journal for 3df9cf7b9785edd3a0db773c7f6b886b: 2023-04-28 14:57:59,593 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(147): Closed 3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:59,593 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=3df9cf7b9785edd3a0db773c7f6b886b, regionState=CLOSED 2023-04-28 14:57:59,594 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693879593"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682693879593"}]},"ts":"1682693879593"} 2023-04-28 14:57:59,596 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-28 14:57:59,596 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 3df9cf7b9785edd3a0db773c7f6b886b, server=jenkins-hbase17.apache.org,37181,1682693812482 in 1.9380 sec 2023-04-28 14:57:59,597 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-28 14:57:59,597 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3df9cf7b9785edd3a0db773c7f6b886b, UNASSIGN in 1.9430 sec 2023-04-28 14:57:59,597 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682693879597"}]},"ts":"1682693879597"} 2023-04-28 14:57:59,598 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-28 14:57:59,599 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-28 14:57:59,601 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9520 sec 2023-04-28 14:57:59,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-28 14:57:59,756 INFO [Listener at localhost.localdomain/40193] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-28 14:57:59,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.HMaster$5(2228): Client=jenkins//136.243.18.41 delete TestAcidGuarantees 2023-04-28 14:57:59,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:57:59,759 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:57:59,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-28 14:57:59,760 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:57:59,764 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:59,767 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A, FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B, FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C, FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/recovered.edits] 2023-04-28 14:57:59,769 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/0e2f1ff12720403db7f7c08be8f22bb8 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/0e2f1ff12720403db7f7c08be8f22bb8 2023-04-28 14:57:59,770 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/62d09bd3353945ef9ce07bc7b21e3c34 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/A/62d09bd3353945ef9ce07bc7b21e3c34 2023-04-28 14:57:59,772 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/0baf4d7fcb844919b44a7b596db306c5 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/0baf4d7fcb844919b44a7b596db306c5 2023-04-28 14:57:59,773 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/7f18b3df7ea5463aa34917f2de0d8e5a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/B/7f18b3df7ea5463aa34917f2de0d8e5a 2023-04-28 14:57:59,775 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/1f326cf1a5c64fc39e60102da65b4bf2 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/1f326cf1a5c64fc39e60102da65b4bf2 2023-04-28 14:57:59,776 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/db40a0fc6aa1437db02b6736cc8e5a2e to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/C/db40a0fc6aa1437db02b6736cc8e5a2e 2023-04-28 14:57:59,778 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/recovered.edits/982.seqid to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b/recovered.edits/982.seqid 2023-04-28 14:57:59,780 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/3df9cf7b9785edd3a0db773c7f6b886b 2023-04-28 14:57:59,780 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 14:57:59,781 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-28 14:57:59,781 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-28 14:57:59,785 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042802cb1a2c068c4b79b9a364d6a552afe3 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042802cb1a2c068c4b79b9a364d6a552afe3 2023-04-28 14:57:59,786 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304280c04dfe803d74c92b4af6d8b28478b54 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304280c04dfe803d74c92b4af6d8b28478b54 2023-04-28 14:57:59,788 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304280c0fc21912ed4c7da6fe2a4cb099ff4f to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304280c0fc21912ed4c7da6fe2a4cb099ff4f 2023-04-28 14:57:59,789 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042822c5076e0c39483aaba81433cb9ca129 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042822c5076e0c39483aaba81433cb9ca129 2023-04-28 14:57:59,790 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042829b76dd665e34f198c974aef86cca314 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042829b76dd665e34f198c974aef86cca314 2023-04-28 14:57:59,791 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304283b8458e0ba374d4b960abee53dc7d08c to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304283b8458e0ba374d4b960abee53dc7d08c 2023-04-28 14:57:59,792 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304283fab8fe867aa40a4aa5d1a39fad6eb4b to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304283fab8fe867aa40a4aa5d1a39fad6eb4b 2023-04-28 14:57:59,793 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042844e6aaec71cb456e946785bcaf6bb2be to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042844e6aaec71cb456e946785bcaf6bb2be 2023-04-28 14:57:59,794 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042855dee64cc3bb449384808086f42a7b24 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042855dee64cc3bb449384808086f42a7b24 2023-04-28 14:57:59,795 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304285b836af016074589815136e1560b1eca to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304285b836af016074589815136e1560b1eca 2023-04-28 14:57:59,801 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428652df8b81c9a4e9bb77c138fd431ca75 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428652df8b81c9a4e9bb77c138fd431ca75 2023-04-28 14:57:59,802 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042874669d17751743738e75729c2ad5824b to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042874669d17751743738e75729c2ad5824b 2023-04-28 14:57:59,803 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304288ecc9b777ff7481689450a37d45a940c to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304288ecc9b777ff7481689450a37d45a940c 2023-04-28 14:57:59,805 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304289d501027105144508129b8f78a0da224 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304289d501027105144508129b8f78a0da224 2023-04-28 14:57:59,806 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428aef9a07c4dd54475b20b5f2f8f844e74 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428aef9a07c4dd54475b20b5f2f8f844e74 2023-04-28 14:57:59,807 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428cd70b5dbff964839aa5ed31bd7c13b29 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428cd70b5dbff964839aa5ed31bd7c13b29 2023-04-28 14:57:59,809 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428cf2a6528766b4c799b4cd14eae1da092 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428cf2a6528766b4c799b4cd14eae1da092 2023-04-28 14:57:59,810 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428d167123328c346ef92b54463c86bf538 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428d167123328c346ef92b54463c86bf538 2023-04-28 14:57:59,811 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428e656403c12d941b197ca7e64d9750dd6 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428e656403c12d941b197ca7e64d9750dd6 2023-04-28 14:57:59,812 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428f8318b5a60314c5fb54d5b5cc928fa70 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428f8318b5a60314c5fb54d5b5cc928fa70 2023-04-28 14:57:59,813 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ff1bd1fa453b4eb082b51c57bf64075e to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ff1bd1fa453b4eb082b51c57bf64075e 2023-04-28 14:57:59,814 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-28 14:57:59,816 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:57:59,819 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-28 14:57:59,824 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-28 14:57:59,826 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:57:59,826 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-28 14:57:59,826 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682693879826"}]},"ts":"9223372036854775807"} 2023-04-28 14:57:59,828 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-28 14:57:59,828 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 3df9cf7b9785edd3a0db773c7f6b886b, NAME => 'TestAcidGuarantees,,1682693843601.3df9cf7b9785edd3a0db773c7f6b886b.', STARTKEY => '', ENDKEY => ''}] 2023-04-28 14:57:59,828 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-28 14:57:59,828 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682693879828"}]},"ts":"9223372036854775807"} 2023-04-28 14:57:59,829 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-28 14:57:59,831 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:57:59,832 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 74 msec 2023-04-28 14:57:59,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-28 14:57:59,861 INFO [Listener at localhost.localdomain/40193] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-28 14:57:59,873 INFO [Listener at localhost.localdomain/40193] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=224 (was 235), OpenFileDescriptor=501 (was 553), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=332 (was 353), ProcessCount=168 (was 168), AvailableMemoryMB=5087 (was 5853) 2023-04-28 14:57:59,884 INFO [Listener at localhost.localdomain/40193] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224, OpenFileDescriptor=501, MaxFileDescriptor=60000, SystemLoadAverage=332, ProcessCount=168, AvailableMemoryMB=5085 2023-04-28 14:57:59,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-28 14:57:59,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.HMaster$4(2112): Client=jenkins//136.243.18.41 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-28 14:57:59,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-28 14:57:59,889 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-28 14:57:59,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(697): Client=jenkins//136.243.18.41 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-28 14:57:59,890 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-28 14:57:59,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-28 14:57:59,891 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8 2023-04-28 14:57:59,892 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8 empty. 2023-04-28 14:57:59,892 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8 2023-04-28 14:57:59,892 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 14:57:59,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-28 14:58:00,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-28 14:58:00,300 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-28 14:58:00,302 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => f40347916069de574ae857f0751743f8, NAME => 'TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp 2023-04-28 14:58:00,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-28 14:58:00,715 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 14:58:00,716 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing f40347916069de574ae857f0751743f8, disabling compactions & flushes 2023-04-28 14:58:00,716 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:00,716 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:00,716 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. after waiting 0 ms 2023-04-28 14:58:00,716 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:00,716 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:00,716 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:00,723 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-28 14:58:00,725 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693880724"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682693880724"}]},"ts":"1682693880724"} 2023-04-28 14:58:00,726 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-28 14:58:00,726 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-28 14:58:00,726 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682693880726"}]},"ts":"1682693880726"} 2023-04-28 14:58:00,727 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-28 14:58:00,730 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f40347916069de574ae857f0751743f8, ASSIGN}] 2023-04-28 14:58:00,731 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f40347916069de574ae857f0751743f8, ASSIGN 2023-04-28 14:58:00,732 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=f40347916069de574ae857f0751743f8, ASSIGN; state=OFFLINE, location=jenkins-hbase17.apache.org,37181,1682693812482; forceNewPlan=false, retain=false 2023-04-28 14:58:00,884 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=f40347916069de574ae857f0751743f8, regionState=OPENING, regionLocation=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:00,884 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693880884"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682693880884"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682693880884"}]},"ts":"1682693880884"} 2023-04-28 14:58:00,888 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482}] 2023-04-28 14:58:00,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-28 14:58:01,051 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:01,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f40347916069de574ae857f0751743f8, NAME => 'TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.', STARTKEY => '', ENDKEY => ''} 2023-04-28 14:58:01,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees f40347916069de574ae857f0751743f8 2023-04-28 14:58:01,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 14:58:01,052 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7894): checking encryption for f40347916069de574ae857f0751743f8 2023-04-28 14:58:01,052 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7897): checking classloading for f40347916069de574ae857f0751743f8 2023-04-28 14:58:01,053 INFO [StoreOpener-f40347916069de574ae857f0751743f8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region f40347916069de574ae857f0751743f8 2023-04-28 14:58:01,056 DEBUG [StoreOpener-f40347916069de574ae857f0751743f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A 2023-04-28 14:58:01,056 DEBUG [StoreOpener-f40347916069de574ae857f0751743f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A 2023-04-28 14:58:01,056 INFO [StoreOpener-f40347916069de574ae857f0751743f8-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 14:58:01,056 INFO [StoreOpener-f40347916069de574ae857f0751743f8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f40347916069de574ae857f0751743f8 columnFamilyName A 2023-04-28 14:58:01,057 INFO [StoreOpener-f40347916069de574ae857f0751743f8-1] regionserver.HStore(310): Store=f40347916069de574ae857f0751743f8/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:58:01,057 INFO [StoreOpener-f40347916069de574ae857f0751743f8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region f40347916069de574ae857f0751743f8 2023-04-28 14:58:01,058 DEBUG [StoreOpener-f40347916069de574ae857f0751743f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B 2023-04-28 14:58:01,058 DEBUG [StoreOpener-f40347916069de574ae857f0751743f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B 2023-04-28 14:58:01,058 INFO [StoreOpener-f40347916069de574ae857f0751743f8-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 14:58:01,058 INFO [StoreOpener-f40347916069de574ae857f0751743f8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f40347916069de574ae857f0751743f8 columnFamilyName B 2023-04-28 14:58:01,059 INFO [StoreOpener-f40347916069de574ae857f0751743f8-1] regionserver.HStore(310): Store=f40347916069de574ae857f0751743f8/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:58:01,059 INFO [StoreOpener-f40347916069de574ae857f0751743f8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region f40347916069de574ae857f0751743f8 2023-04-28 14:58:01,060 DEBUG [StoreOpener-f40347916069de574ae857f0751743f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C 2023-04-28 14:58:01,060 DEBUG [StoreOpener-f40347916069de574ae857f0751743f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C 2023-04-28 14:58:01,060 INFO [StoreOpener-f40347916069de574ae857f0751743f8-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 14:58:01,060 INFO [StoreOpener-f40347916069de574ae857f0751743f8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f40347916069de574ae857f0751743f8 columnFamilyName C 2023-04-28 14:58:01,060 INFO [StoreOpener-f40347916069de574ae857f0751743f8-1] regionserver.HStore(310): Store=f40347916069de574ae857f0751743f8/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:58:01,060 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:01,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8 2023-04-28 14:58:01,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8 2023-04-28 14:58:01,063 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 14:58:01,064 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1055): writing seq id for f40347916069de574ae857f0751743f8 2023-04-28 14:58:01,065 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-28 14:58:01,066 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1072): Opened f40347916069de574ae857f0751743f8; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71493086, jitterRate=0.06532999873161316}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 14:58:01,066 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(965): Region open journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:01,066 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8., pid=30, masterSystemTime=1682693881042 2023-04-28 14:58:01,068 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:01,068 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:01,068 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=f40347916069de574ae857f0751743f8, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:01,068 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693881068"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1682693881068"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682693881068"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682693881068"}]},"ts":"1682693881068"} 2023-04-28 14:58:01,071 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-28 14:58:01,071 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 in 181 msec 2023-04-28 14:58:01,072 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-28 14:58:01,072 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=f40347916069de574ae857f0751743f8, ASSIGN in 341 msec 2023-04-28 14:58:01,073 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-28 14:58:01,073 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682693881073"}]},"ts":"1682693881073"} 2023-04-28 14:58:01,073 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-28 14:58:01,075 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-28 14:58:01,076 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1880 sec 2023-04-28 14:58:01,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-28 14:58:01,999 INFO [Listener at localhost.localdomain/40193] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-28 14:58:02,001 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x5c466b6c to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:58:02,010 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@12394d27, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:58:02,011 DEBUG [Listener at localhost.localdomain/40193] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:58:02,013 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:50208, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:58:02,014 DEBUG [Listener at localhost.localdomain/40193] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-28 14:58:02,017 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:36810, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-28 14:58:02,019 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x46d74611 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:58:02,024 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29a548ac, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:58:02,024 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x3f837278 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:58:02,034 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43184c57, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:58:02,034 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x5a8a575d to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:58:02,038 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e3efec6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:58:02,038 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x6347f2e6 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:58:02,041 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6b7f716c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:58:02,042 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x36523181 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:58:02,045 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ade356f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:58:02,046 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x4cbc74e7 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:58:02,056 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@57d25552, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:58:02,056 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x3c5d1710 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:58:02,061 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13695730, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:58:02,062 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x5d22a55e to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:58:02,065 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@993cc7d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:58:02,065 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x2179ead9 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:58:02,070 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69b9b3fd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:58:02,070 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x7897b06a to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:58:02,074 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1656b283, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:58:02,084 DEBUG [hconnection-0x4ce49e0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:58:02,084 DEBUG [hconnection-0x477a9640-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:58:02,084 DEBUG [hconnection-0x7ffc665-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:58:02,084 DEBUG [hconnection-0x341f394d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:58:02,084 DEBUG [hconnection-0xd44e959-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:58:02,084 DEBUG [hconnection-0xd03038e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:58:02,084 DEBUG [hconnection-0x5c989513-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:58:02,085 DEBUG [hconnection-0x6f590bb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:58:02,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:02,086 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:50228, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:58:02,086 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:50268, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:58:02,088 DEBUG [hconnection-0x5555d87-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:58:02,088 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:50260, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:58:02,088 DEBUG [hconnection-0x7f1c5530-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:58:02,088 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:50216, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:58:02,090 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:50276, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:58:02,090 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:50218, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:58:02,090 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:50274, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:58:02,091 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:50244, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:58:02,091 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:50270, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:58:02,094 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:50300, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:58:02,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:02,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:02,095 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:02,095 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:02,095 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:02,095 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:02,095 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,095 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:02,096 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:02,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:02,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:02,096 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:02,096 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,096 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:02,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:02,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:02,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:02,097 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:02,097 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:02,097 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:02,097 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:02,097 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:02,097 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:02,097 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:02,097 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:02,098 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:02,103 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:02,104 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:02,104 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:02,104 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:02,104 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,104 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:02,104 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:02,105 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,105 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:02,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:02,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:02,107 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,107 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:02,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:02,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:02,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:02,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:02,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2333c09e[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:02,108 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:02,108 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,109 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,109 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:02,109 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,109 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:02,109 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:02,109 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:02,110 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,110 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:02,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:02,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:02,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on f40347916069de574ae857f0751743f8 2023-04-28 14:58:02,110 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f40347916069de574ae857f0751743f8 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-28 14:58:02,110 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:02,110 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:02,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:02,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:02,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:02,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:02,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:02,112 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=A 2023-04-28 14:58:02,112 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:02,112 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=B 2023-04-28 14:58:02,112 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:02,112 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=C 2023-04-28 14:58:02,112 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:02,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:02,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:02,113 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:02,113 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:02,113 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:02,113 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:02,113 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:02,117 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,117 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:02,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:02,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,117 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:02,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:02,118 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:02,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:02,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:02,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:02,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:02,119 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:02,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:02,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,122 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,122 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:02,122 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:02,122 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:02,122 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:02,122 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:02,122 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,122 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,122 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,122 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:02,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:02,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:02,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:02,123 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:02,123 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:02,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:02,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:02,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:02,130 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/70d377b4e88a4cc3adbd1d7b18ba121a 2023-04-28 14:58:02,130 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:02,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693942128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:02,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693942128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:02,131 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:02,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693942128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50270 deadline: 1682693942129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:02,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693942129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,223 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:02,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:02,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:02,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:02,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:02,225 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:02,225 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:02,225 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:02,225 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:02,226 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,226 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:02,226 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,226 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:02,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:02,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:02,227 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,227 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:02,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:02,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:02,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:02,227 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:02,227 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:02,227 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:02,228 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:02,228 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:02,228 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:02,228 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:02,228 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:02,228 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:02,228 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:02,228 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:02,228 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:02,228 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:02,228 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:02,228 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,228 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:02,228 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:02,229 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,229 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:02,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:02,229 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,229 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:02,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:02,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:02,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:02,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:02,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:02,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4aebc039[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:02,230 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:02,230 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,231 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,231 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,231 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:02,231 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:02,231 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:02,231 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:02,232 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:02,232 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,232 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:02,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:02,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:02,232 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:02,232 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:02,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:02,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693942231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,232 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:02,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693942231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:02,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:02,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:02,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:02,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:02,234 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:02,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693942232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50270 deadline: 1682693942232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:02,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:02,234 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:02,234 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:02,234 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:02,234 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:02,234 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:02,235 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:02,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:02,235 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,235 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:02,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:02,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:02,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:02,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:02,236 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:02,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:02,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:02,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693942234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,237 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:02,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:02,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,238 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,238 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:02,238 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:02,238 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:02,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:02,238 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:02,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:02,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:02,238 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:02,238 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:02,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:02,238 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:02,238 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:02,238 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,238 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,239 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:02,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:02,338 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:02,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:02,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:02,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:02,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:02,341 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:02,341 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:02,341 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:02,341 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:02,342 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,342 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:02,342 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,342 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:02,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:02,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:02,342 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,342 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:02,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:02,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:02,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:02,343 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:02,343 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:02,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:02,344 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:02,344 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:02,344 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:02,344 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:02,344 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:02,344 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:02,344 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:02,344 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:02,344 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:02,344 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:02,344 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:02,344 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,344 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:02,344 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:02,345 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,345 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:02,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:02,345 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,345 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:02,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:02,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:02,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:02,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:02,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:02,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6103e591[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:02,346 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:02,346 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,347 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,347 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,347 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:02,347 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:02,347 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:02,347 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:02,348 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,348 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:02,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:02,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:02,348 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:02,348 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:02,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:02,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:02,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:02,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:02,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:02,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:02,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:02,350 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:02,350 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:02,350 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:02,350 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:02,350 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:02,351 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,351 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:02,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:02,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,351 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:02,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:02,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,351 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:02,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:02,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:02,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:02,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:02,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,352 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:02,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:02,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,354 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,354 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:02,354 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:02,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:02,354 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:02,354 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:02,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:02,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:02,354 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:02,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:02,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:02,354 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,354 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,354 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,354 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,354 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:02,354 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:02,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:02,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:02,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:02,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693942434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:02,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693942434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:02,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693942435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:02,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50270 deadline: 1682693942436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,438 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:02,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693942437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,454 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:02,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:02,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:02,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:02,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:02,457 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:02,457 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:02,457 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:02,457 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:02,457 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,457 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:02,458 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:02,458 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:02,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:02,458 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,458 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:02,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:02,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:02,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:02,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:02,461 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:02,461 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:02,461 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:02,461 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:02,461 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:02,462 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:02,462 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:02,462 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:02,462 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:02,462 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:02,462 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:02,462 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:02,462 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:02,462 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,462 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:02,462 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:02,463 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:02,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:02,463 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:02,464 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,464 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:02,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:02,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:02,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:02,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:02,465 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:02,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@406e4000[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:02,465 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,465 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,465 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:02,465 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:02,465 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:02,465 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,465 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:02,466 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:02,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:02,466 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:02,466 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:02,466 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:02,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:02,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:02,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:02,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:02,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:02,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:02,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:02,468 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:02,468 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:02,468 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:02,468 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:02,468 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:02,469 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,469 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,469 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:02,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:02,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:02,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:02,469 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:02,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:02,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:02,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:02,470 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:02,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:02,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:02,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,472 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,472 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:02,472 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:02,472 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:02,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:02,472 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:02,472 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:02,472 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:02,472 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:02,472 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:02,472 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:02,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:02,472 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:02,473 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:02,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:02,562 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/49cdd87bb5c047e384902806075cba30 2023-04-28 14:58:02,573 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:02,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:02,574 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/bb7a797e4df14b8fb8a452aeb40ef9dc 2023-04-28 14:58:02,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:02,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:02,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:02,576 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:02,576 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:02,576 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:02,576 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:02,576 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,576 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:02,577 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,577 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:02,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:02,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:02,577 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:02,577 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:02,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,578 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/70d377b4e88a4cc3adbd1d7b18ba121a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/70d377b4e88a4cc3adbd1d7b18ba121a 2023-04-28 14:58:02,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:02,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:02,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:02,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:02,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:02,579 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:02,579 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:02,579 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:02,580 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:02,580 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:02,580 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:02,580 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:02,580 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:02,580 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:02,580 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:02,580 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:02,580 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,580 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:02,580 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:02,581 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,581 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:02,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:02,582 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,582 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:02,582 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/70d377b4e88a4cc3adbd1d7b18ba121a, entries=150, sequenceid=15, filesize=11.6 K 2023-04-28 14:58:02,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:02,583 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/49cdd87bb5c047e384902806075cba30 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/49cdd87bb5c047e384902806075cba30 2023-04-28 14:58:02,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:02,583 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:02,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7de45fa3[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:02,584 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,584 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,584 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:02,585 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:02,585 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:02,585 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,585 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:02,585 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:02,585 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,586 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:02,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:02,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:02,586 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:02,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:02,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:02,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:02,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:02,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:02,588 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/49cdd87bb5c047e384902806075cba30, entries=150, sequenceid=15, filesize=11.6 K 2023-04-28 14:58:02,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:02,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:02,589 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:02,589 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:02,589 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:02,589 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/bb7a797e4df14b8fb8a452aeb40ef9dc as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/bb7a797e4df14b8fb8a452aeb40ef9dc 2023-04-28 14:58:02,589 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:02,589 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:02,591 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,591 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:02,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,591 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:02,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:02,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:02,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:02,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,592 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:02,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:02,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:02,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:02,593 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:02,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:02,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,595 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:02,595 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:02,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:02,595 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:02,595 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:02,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:02,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:02,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:02,595 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:02,595 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:02,596 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,596 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:02,596 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,596 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:02,596 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:02,596 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:02,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:02,598 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/bb7a797e4df14b8fb8a452aeb40ef9dc, entries=150, sequenceid=15, filesize=11.6 K 2023-04-28 14:58:02,598 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for f40347916069de574ae857f0751743f8 in 488ms, sequenceid=15, compaction requested=false 2023-04-28 14:58:02,599 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-28 14:58:02,599 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:02,696 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:02,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:02,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:02,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:02,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:02,699 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:02,699 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:02,699 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:02,699 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:02,699 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,699 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:02,700 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:02,700 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:02,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:02,700 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:02,700 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:02,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:02,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:02,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:02,700 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:02,701 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:02,701 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:02,701 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:02,701 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:02,701 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:02,701 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:02,701 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:02,701 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:02,701 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:02,701 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f40347916069de574ae857f0751743f8 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-28 14:58:02,701 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=A 2023-04-28 14:58:02,701 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:02,701 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=B 2023-04-28 14:58:02,701 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:02,701 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=C 2023-04-28 14:58:02,702 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:02,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on f40347916069de574ae857f0751743f8 2023-04-28 14:58:02,739 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:02,752 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:02,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693942749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,752 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:02,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693942749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:02,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693942751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:02,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693942751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,755 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:02,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50270 deadline: 1682693942752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,854 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:02,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693942853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,854 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:02,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693942853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:02,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693942854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:02,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693942855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:02,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:02,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50270 deadline: 1682693942856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:03,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:03,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693943055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:03,056 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:03,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693943056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:03,058 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:03,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693943056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:03,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:03,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693943058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:03,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:03,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50270 deadline: 1682693943058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:03,108 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/c9c6d4e6f7ba4c62ad47067533b6baf6 2023-04-28 14:58:03,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:03,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693943357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:03,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:03,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693943358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:03,362 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:03,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693943361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:03,362 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:03,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693943361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:03,364 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:03,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50270 deadline: 1682693943362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:03,516 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/08423da28b5c4b418e07807ad15ae8b7 2023-04-28 14:58:03,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:03,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693943862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:03,863 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:03,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693943863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:03,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:03,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693943864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:03,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:03,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693943865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:03,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:03,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50270 deadline: 1682693943867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:03,926 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/47c6c7e4f12941a89435f77142b14f7d 2023-04-28 14:58:03,929 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/c9c6d4e6f7ba4c62ad47067533b6baf6 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/c9c6d4e6f7ba4c62ad47067533b6baf6 2023-04-28 14:58:03,932 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/c9c6d4e6f7ba4c62ad47067533b6baf6, entries=150, sequenceid=34, filesize=11.6 K 2023-04-28 14:58:03,933 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/08423da28b5c4b418e07807ad15ae8b7 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/08423da28b5c4b418e07807ad15ae8b7 2023-04-28 14:58:03,936 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/08423da28b5c4b418e07807ad15ae8b7, entries=150, sequenceid=34, filesize=11.6 K 2023-04-28 14:58:03,938 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/47c6c7e4f12941a89435f77142b14f7d as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/47c6c7e4f12941a89435f77142b14f7d 2023-04-28 14:58:03,943 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/47c6c7e4f12941a89435f77142b14f7d, entries=150, sequenceid=34, filesize=11.6 K 2023-04-28 14:58:03,944 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for f40347916069de574ae857f0751743f8 in 1243ms, sequenceid=34, compaction requested=false 2023-04-28 14:58:03,944 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:03,944 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:03,944 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:03,945 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:03,945 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:03,945 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:03,945 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:03,945 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:03,945 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:03,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:03,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:03,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:03,946 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:03,946 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:03,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:03,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:03,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:03,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:03,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:03,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:03,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@691ab22e[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:03,947 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:03,947 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:03,947 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:03,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:03,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:03,947 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:03,947 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:03,948 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:03,948 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:03,948 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:03,948 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:03,948 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:03,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:03,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:03,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:03,948 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:03,948 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:03,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:03,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:03,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:03,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:03,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:03,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:03,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:03,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:03,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:03,950 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:03,950 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:03,950 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:03,950 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:03,950 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:03,951 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:03,951 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:03,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:03,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:03,951 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:03,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:03,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:03,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:03,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:03,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:03,952 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:03,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:03,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:03,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:03,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:03,953 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:03,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:03,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:03,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:03,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:03,954 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:03,954 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:03,954 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:03,954 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:03,954 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:03,954 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:03,954 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:03,954 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:03,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:03,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:03,955 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:03,955 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:03,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:03,955 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:03,955 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:03,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:03,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:03,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:03,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:04,055 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:04,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:04,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:04,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:04,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:04,058 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:04,058 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:04,058 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:04,058 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:04,058 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:04,058 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:04,059 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:04,059 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:04,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:04,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:04,059 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:04,059 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:04,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:04,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:04,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:04,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:04,060 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:04,060 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:04,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:04,060 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:04,060 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:04,060 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:04,060 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:04,060 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:04,060 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:04,060 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:04,061 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f40347916069de574ae857f0751743f8 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-28 14:58:04,061 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=A 2023-04-28 14:58:04,061 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:04,061 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=B 2023-04-28 14:58:04,061 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:04,061 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=C 2023-04-28 14:58:04,061 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:04,068 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/f156cfd423824719b685280a380f831a 2023-04-28 14:58:04,496 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/fdd97e813e574ae79762ebc48678fb41 2023-04-28 14:58:04,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on f40347916069de574ae857f0751743f8 2023-04-28 14:58:04,865 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:04,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:04,879 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:04,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693944874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:04,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693944875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:04,880 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:04,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693944877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:04,880 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:04,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50270 deadline: 1682693944878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:04,881 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:04,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693944879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:04,905 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/067cced26b5343c59bca5e0fdbb1366d 2023-04-28 14:58:04,909 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/f156cfd423824719b685280a380f831a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/f156cfd423824719b685280a380f831a 2023-04-28 14:58:04,912 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/f156cfd423824719b685280a380f831a, entries=150, sequenceid=47, filesize=11.6 K 2023-04-28 14:58:04,913 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/fdd97e813e574ae79762ebc48678fb41 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/fdd97e813e574ae79762ebc48678fb41 2023-04-28 14:58:04,918 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/fdd97e813e574ae79762ebc48678fb41, entries=150, sequenceid=47, filesize=11.6 K 2023-04-28 14:58:04,919 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/067cced26b5343c59bca5e0fdbb1366d as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/067cced26b5343c59bca5e0fdbb1366d 2023-04-28 14:58:04,930 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/067cced26b5343c59bca5e0fdbb1366d, entries=150, sequenceid=47, filesize=11.6 K 2023-04-28 14:58:04,930 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for f40347916069de574ae857f0751743f8 in 870ms, sequenceid=47, compaction requested=true 2023-04-28 14:58:04,930 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:04,930 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:04,931 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:04,931 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:04,931 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:04,931 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:04,931 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:04,932 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:04,932 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:04,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:04,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:04,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:04,932 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:04,932 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:04,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:04,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:04,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:04,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:04,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:04,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:04,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@174bfdf1[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:04,934 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:04,934 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:04,935 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:04,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:04,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:04,935 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:04,935 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:04,935 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:04,935 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:04,935 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:04,936 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:04,936 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:04,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:04,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:04,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:04,936 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:04,936 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:04,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:04,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:04,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:04,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:04,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:04,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:04,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:04,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:04,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:04,939 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:04,939 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:04,939 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:04,939 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:04,939 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:04,940 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:04,940 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:04,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:04,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:04,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:04,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:04,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:04,940 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:04,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:04,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:04,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:04,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:04,940 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:04,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:04,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:04,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:04,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:04,942 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:04,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:04,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:04,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:04,944 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:04,944 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:04,944 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:04,944 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:04,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:04,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:04,944 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:04,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:04,944 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:04,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:04,944 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:04,944 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:04,944 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:04,944 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:04,944 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:04,944 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:04,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:04,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:04,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on f40347916069de574ae857f0751743f8 2023-04-28 14:58:04,983 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f40347916069de574ae857f0751743f8 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-28 14:58:04,984 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=A 2023-04-28 14:58:04,984 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:04,984 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=B 2023-04-28 14:58:04,984 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:04,984 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=C 2023-04-28 14:58:04,984 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:04,993 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:04,993 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/7b5b284ced40412681ecd698d1098760 2023-04-28 14:58:04,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693944988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:04,995 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:04,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693944990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:04,995 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:04,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693944990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:04,995 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:04,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50270 deadline: 1682693944990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:04,996 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:04,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693944992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,005 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/3c8599f522ab4c278f8a691fe519fe30 2023-04-28 14:58:05,044 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:05,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:05,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:05,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:05,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:05,047 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:05,047 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:05,047 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:05,047 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:05,047 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,047 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:05,049 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,049 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:05,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:05,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:05,049 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,049 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:05,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:05,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:05,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:05,050 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:05,050 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:05,050 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:05,050 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:05,050 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:05,050 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:05,050 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:05,050 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:05,050 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:05,050 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:05,050 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:05,050 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:05,050 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:05,050 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:05,050 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,050 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:05,050 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:05,051 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,051 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:05,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:05,052 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,052 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:05,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:05,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:05,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:05,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:05,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:05,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2597d158[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:05,053 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:05,053 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,053 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,054 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,054 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:05,054 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:05,054 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:05,054 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:05,054 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,054 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:05,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:05,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:05,054 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:05,056 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:05,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:05,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:05,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:05,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:05,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:05,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:05,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:05,058 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:05,058 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:05,058 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:05,058 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:05,059 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:05,059 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,059 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:05,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:05,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,059 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:05,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:05,060 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:05,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:05,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:05,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:05,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:05,061 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:05,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:05,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,062 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,062 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:05,062 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:05,062 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,062 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,062 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:05,062 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:05,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:05,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:05,063 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:05,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:05,063 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:05,063 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:05,063 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:05,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:05,063 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:05,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:05,096 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:05,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693945094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:05,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693945096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:05,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693945096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:05,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50270 deadline: 1682693945096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:05,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693945096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,122 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-28 14:58:05,163 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:05,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:05,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:05,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:05,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:05,166 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:05,166 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:05,166 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:05,166 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:05,166 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,166 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:05,167 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,167 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:05,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:05,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:05,167 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,167 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:05,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:05,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:05,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:05,168 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:05,168 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:05,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:05,168 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:05,168 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:05,168 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:05,168 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:05,168 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:05,168 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:05,168 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:05,168 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:05,168 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:05,168 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:05,168 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:05,168 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,169 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:05,169 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:05,169 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,169 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:05,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:05,170 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,170 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:05,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:05,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:05,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:05,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:05,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:05,171 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:05,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ad19b0e[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:05,171 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,171 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,171 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:05,172 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:05,172 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:05,172 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,172 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:05,172 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:05,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:05,173 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:05,173 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:05,173 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:05,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:05,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:05,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:05,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:05,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:05,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:05,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:05,174 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:05,174 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:05,174 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:05,175 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:05,175 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:05,175 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,175 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:05,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:05,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,175 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:05,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:05,176 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:05,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:05,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:05,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:05,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:05,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,177 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:05,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:05,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,179 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,179 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:05,179 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:05,179 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,179 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,179 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:05,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:05,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:05,179 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,179 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:05,179 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:05,179 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:05,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:05,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:05,179 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:05,179 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:05,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:05,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:05,279 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:05,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:05,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:05,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:05,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:05,282 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:05,282 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:05,282 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:05,282 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:05,283 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,283 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:05,283 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:05,283 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:05,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:05,284 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,284 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:05,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:05,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:05,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:05,284 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:05,284 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:05,284 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:05,284 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:05,285 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:05,285 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:05,285 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:05,285 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:05,285 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:05,285 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:05,285 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:05,285 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:05,285 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:05,285 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:05,285 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,285 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:05,285 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:05,286 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,286 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:05,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:05,286 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,286 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:05,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:05,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:05,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:05,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:05,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:05,287 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:05,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46e6cdb8[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:05,287 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,288 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,288 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:05,288 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:05,288 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:05,288 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,288 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:05,289 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,289 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:05,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:05,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:05,289 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:05,289 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:05,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:05,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:05,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:05,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:05,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:05,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:05,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:05,291 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:05,291 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:05,291 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:05,291 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:05,291 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:05,292 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,292 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,292 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:05,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:05,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:05,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:05,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:05,293 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:05,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:05,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:05,294 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:05,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:05,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:05,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,295 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,295 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:05,295 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:05,295 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:05,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:05,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:05,295 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:05,295 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,295 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,295 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,295 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,295 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:05,296 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:05,295 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:05,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:05,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:05,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:05,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:05,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:05,299 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:05,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693945298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,299 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:05,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693945299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,301 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:05,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693945299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:05,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693945300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:05,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50270 deadline: 1682693945300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,396 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:05,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:05,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:05,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:05,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:05,398 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:05,398 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:05,399 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:05,399 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:05,399 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,399 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:05,400 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:05,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:05,400 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:05,400 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,400 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:05,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:05,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:05,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:05,400 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:05,401 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:05,401 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:05,401 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:05,401 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:05,401 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:05,401 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:05,401 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:05,401 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:05,401 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:05,401 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:05,401 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:05,401 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:05,401 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:05,401 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,401 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:05,401 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:05,402 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,402 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:05,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:05,403 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,403 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:05,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:05,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:05,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:05,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:05,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:05,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e0a14e0[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:05,404 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:05,404 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,404 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,405 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:05,405 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:05,405 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:05,405 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,405 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:05,405 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,405 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:05,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:05,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:05,405 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:05,405 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:05,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:05,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:05,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:05,408 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:05,408 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:05,408 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:05,408 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:05,408 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:05,409 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,409 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:05,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:05,409 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:05,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:05,409 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:05,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:05,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:05,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:05,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:05,410 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:05,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:05,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,411 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,412 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:05,412 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:05,412 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:05,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:05,412 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:05,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:05,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:05,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:05,412 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:05,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:05,412 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,412 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,412 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:05,412 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:05,412 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:05,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:05,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:05,412 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,415 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/2522d83b91f8482587078da78437f824 2023-04-28 14:58:05,418 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/7b5b284ced40412681ecd698d1098760 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/7b5b284ced40412681ecd698d1098760 2023-04-28 14:58:05,421 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/7b5b284ced40412681ecd698d1098760, entries=200, sequenceid=70, filesize=13.9 K 2023-04-28 14:58:05,422 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/3c8599f522ab4c278f8a691fe519fe30 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/3c8599f522ab4c278f8a691fe519fe30 2023-04-28 14:58:05,425 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/3c8599f522ab4c278f8a691fe519fe30, entries=150, sequenceid=70, filesize=11.6 K 2023-04-28 14:58:05,427 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/2522d83b91f8482587078da78437f824 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/2522d83b91f8482587078da78437f824 2023-04-28 14:58:05,432 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/2522d83b91f8482587078da78437f824, entries=150, sequenceid=70, filesize=11.6 K 2023-04-28 14:58:05,432 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~399.84 KB/409440, currentSize=40.25 KB/41220 for f40347916069de574ae857f0751743f8 in 449ms, sequenceid=70, compaction requested=true 2023-04-28 14:58:05,432 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:05,432 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:05,432 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:05,432 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:58:05,432 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:05,432 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:58:05,435 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49884 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:58:05,435 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): f40347916069de574ae857f0751743f8/A is initiating minor compaction (all files) 2023-04-28 14:58:05,435 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:58:05,435 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f40347916069de574ae857f0751743f8/A in TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:05,435 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): f40347916069de574ae857f0751743f8/B is initiating minor compaction (all files) 2023-04-28 14:58:05,435 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f40347916069de574ae857f0751743f8/B in TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:05,435 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/70d377b4e88a4cc3adbd1d7b18ba121a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/c9c6d4e6f7ba4c62ad47067533b6baf6, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/f156cfd423824719b685280a380f831a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/7b5b284ced40412681ecd698d1098760] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp, totalSize=48.7 K 2023-04-28 14:58:05,435 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/49cdd87bb5c047e384902806075cba30, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/08423da28b5c4b418e07807ad15ae8b7, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/fdd97e813e574ae79762ebc48678fb41, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/3c8599f522ab4c278f8a691fe519fe30] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp, totalSize=46.4 K 2023-04-28 14:58:05,436 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 70d377b4e88a4cc3adbd1d7b18ba121a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682693882109 2023-04-28 14:58:05,436 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 49cdd87bb5c047e384902806075cba30, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682693882109 2023-04-28 14:58:05,436 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting c9c6d4e6f7ba4c62ad47067533b6baf6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1682693882126 2023-04-28 14:58:05,436 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 08423da28b5c4b418e07807ad15ae8b7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1682693882126 2023-04-28 14:58:05,436 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting f156cfd423824719b685280a380f831a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1682693882741 2023-04-28 14:58:05,436 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting fdd97e813e574ae79762ebc48678fb41, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1682693882741 2023-04-28 14:58:05,436 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 7b5b284ced40412681ecd698d1098760, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682693884869 2023-04-28 14:58:05,436 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 3c8599f522ab4c278f8a691fe519fe30, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682693884869 2023-04-28 14:58:05,443 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): f40347916069de574ae857f0751743f8#B#compaction#192 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:05,443 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): f40347916069de574ae857f0751743f8#A#compaction#193 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:05,472 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/7f2dd54a6d87480383080f0131188611 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/7f2dd54a6d87480383080f0131188611 2023-04-28 14:58:05,472 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/94d8d51e11534d59bf433357f9f830f1 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/94d8d51e11534d59bf433357f9f830f1 2023-04-28 14:58:05,478 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f40347916069de574ae857f0751743f8/B of f40347916069de574ae857f0751743f8 into 7f2dd54a6d87480383080f0131188611(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:05,478 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:05,478 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8., storeName=f40347916069de574ae857f0751743f8/B, priority=12, startTime=1682693885432; duration=0sec 2023-04-28 14:58:05,478 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:05,478 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:58:05,480 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:58:05,480 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): f40347916069de574ae857f0751743f8/C is initiating minor compaction (all files) 2023-04-28 14:58:05,480 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f40347916069de574ae857f0751743f8/C in TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:05,480 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/bb7a797e4df14b8fb8a452aeb40ef9dc, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/47c6c7e4f12941a89435f77142b14f7d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/067cced26b5343c59bca5e0fdbb1366d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/2522d83b91f8482587078da78437f824] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp, totalSize=46.4 K 2023-04-28 14:58:05,480 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting bb7a797e4df14b8fb8a452aeb40ef9dc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682693882109 2023-04-28 14:58:05,481 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 47c6c7e4f12941a89435f77142b14f7d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1682693882126 2023-04-28 14:58:05,481 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 067cced26b5343c59bca5e0fdbb1366d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1682693882741 2023-04-28 14:58:05,481 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 2522d83b91f8482587078da78437f824, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682693884869 2023-04-28 14:58:05,499 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): f40347916069de574ae857f0751743f8#C#compaction#194 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:05,502 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f40347916069de574ae857f0751743f8/A of f40347916069de574ae857f0751743f8 into 94d8d51e11534d59bf433357f9f830f1(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:05,502 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:05,502 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8., storeName=f40347916069de574ae857f0751743f8/A, priority=12, startTime=1682693885432; duration=0sec 2023-04-28 14:58:05,503 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:05,512 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:05,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:05,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:05,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:05,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:05,518 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:05,518 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:05,518 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:05,518 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:05,518 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,518 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:05,519 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:05,519 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:05,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:05,519 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,519 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:05,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:05,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:05,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:05,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:05,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:05,521 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:05,521 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:05,521 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:05,521 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:05,521 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:05,530 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:05,530 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:05,530 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:05,530 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:05,530 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f40347916069de574ae857f0751743f8 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-28 14:58:05,531 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=A 2023-04-28 14:58:05,532 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:05,532 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=B 2023-04-28 14:58:05,532 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:05,532 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=C 2023-04-28 14:58:05,532 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:05,534 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/76a6167544fa43fcb34e87cd498a36c3 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/76a6167544fa43fcb34e87cd498a36c3 2023-04-28 14:58:05,541 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f40347916069de574ae857f0751743f8/C of f40347916069de574ae857f0751743f8 into 76a6167544fa43fcb34e87cd498a36c3(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:05,541 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:05,541 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8., storeName=f40347916069de574ae857f0751743f8/C, priority=12, startTime=1682693885432; duration=0sec 2023-04-28 14:58:05,541 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:05,557 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/64aa34bfcbea4485a9f771c9860e6cef 2023-04-28 14:58:05,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on f40347916069de574ae857f0751743f8 2023-04-28 14:58:05,601 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:05,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682693885605, max timestamp=1682693885614Num uniques 859; 2023-04-28 14:58:05,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 14:58:05,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:05,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:05,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693945619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693945620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,623 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:05,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:05,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693945619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693945620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,623 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:05,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50270 deadline: 1682693945621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,725 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:05,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693945724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,725 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:05,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693945724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,725 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:05,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50270 deadline: 1682693945724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,725 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:05,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693945724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,726 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:05,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693945724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:05,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693945926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:05,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50270 deadline: 1682693945926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:05,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693945926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,928 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:05,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693945926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,928 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:05,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693945927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:05,978 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/f6110f1425ea4ac78a05b41dd3e2137a 2023-04-28 14:58:05,992 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/fcb2840355cd4db49c8d5ea1461284fa 2023-04-28 14:58:05,995 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/64aa34bfcbea4485a9f771c9860e6cef as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/64aa34bfcbea4485a9f771c9860e6cef 2023-04-28 14:58:05,998 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/64aa34bfcbea4485a9f771c9860e6cef, entries=100, sequenceid=81, filesize=9.3 K 2023-04-28 14:58:05,999 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/f6110f1425ea4ac78a05b41dd3e2137a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/f6110f1425ea4ac78a05b41dd3e2137a 2023-04-28 14:58:06,002 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/f6110f1425ea4ac78a05b41dd3e2137a, entries=100, sequenceid=81, filesize=9.3 K 2023-04-28 14:58:06,002 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/fcb2840355cd4db49c8d5ea1461284fa as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/fcb2840355cd4db49c8d5ea1461284fa 2023-04-28 14:58:06,008 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/fcb2840355cd4db49c8d5ea1461284fa, entries=100, sequenceid=81, filesize=9.3 K 2023-04-28 14:58:06,009 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=134.18 KB/137400 for f40347916069de574ae857f0751743f8 in 479ms, sequenceid=81, compaction requested=false 2023-04-28 14:58:06,009 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:06,009 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:06,009 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:06,009 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:06,009 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,009 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:06,009 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:06,010 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,010 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:06,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:06,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:06,011 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:06,011 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:06,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:06,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:06,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:06,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:06,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:06,012 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:06,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@150c1968[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:06,012 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:06,012 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:06,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:06,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:06,013 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:06,013 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:06,013 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,013 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:06,013 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:06,013 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:06,013 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,013 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:06,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:06,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:06,014 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:06,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:06,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:06,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:06,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:06,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:06,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:06,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:06,015 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:06,016 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:06,016 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:06,016 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:06,016 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:06,017 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:06,017 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:06,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:06,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:06,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:06,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:06,017 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:06,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:06,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:06,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:06,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:06,017 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:06,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:06,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:06,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:06,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,018 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:06,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:06,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,020 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,020 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:06,020 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:06,020 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:06,020 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:06,020 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:06,020 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:06,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:06,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:06,020 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:06,020 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:06,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:06,020 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:06,020 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:06,020 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:06,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:06,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:06,121 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:06,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:06,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:06,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:06,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:06,123 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:06,123 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:06,123 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:06,123 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:06,124 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:06,124 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:06,124 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,124 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:06,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:06,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:06,124 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,125 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:06,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:06,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:06,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:06,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:06,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:06,125 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:06,125 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:06,125 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:06,125 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:06,126 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:06,126 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:06,126 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:06,126 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:06,126 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:06,126 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f40347916069de574ae857f0751743f8 3/3 column families, dataSize=134.18 KB heapSize=391.80 KB 2023-04-28 14:58:06,126 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=A 2023-04-28 14:58:06,126 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:06,126 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=B 2023-04-28 14:58:06,126 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:06,126 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=C 2023-04-28 14:58:06,126 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:06,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on f40347916069de574ae857f0751743f8 2023-04-28 14:58:06,229 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:06,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:06,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693946232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:06,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50270 deadline: 1682693946232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:06,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693946233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:06,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693946233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:06,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693946233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:06,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693946335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:06,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50270 deadline: 1682693946336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:06,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693946336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:06,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:06,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693946336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693946336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,536 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/63bc9271a4ee4fe286cf9f7cefef0483 2023-04-28 14:58:06,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:06,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50270 deadline: 1682693946538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:06,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693946539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:06,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693946539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:06,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693946540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:06,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693946540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:06,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50270 deadline: 1682693946841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:06,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693946841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,843 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:06,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693946842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,843 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:06,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693946843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,843 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:06,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693946843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:06,953 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/97ac3ad5d5fc4bd283fd4ef5a5c3200e 2023-04-28 14:58:07,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:07,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693947344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:07,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:07,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693947344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:07,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:07,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50270 deadline: 1682693947345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:07,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:07,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693947346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:07,349 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:07,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693947348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:07,362 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/164cdd2be30e424692f1d6646d33c914 2023-04-28 14:58:07,366 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/63bc9271a4ee4fe286cf9f7cefef0483 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/63bc9271a4ee4fe286cf9f7cefef0483 2023-04-28 14:58:07,368 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/63bc9271a4ee4fe286cf9f7cefef0483, entries=150, sequenceid=109, filesize=11.6 K 2023-04-28 14:58:07,369 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/97ac3ad5d5fc4bd283fd4ef5a5c3200e as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/97ac3ad5d5fc4bd283fd4ef5a5c3200e 2023-04-28 14:58:07,372 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/97ac3ad5d5fc4bd283fd4ef5a5c3200e, entries=150, sequenceid=109, filesize=11.6 K 2023-04-28 14:58:07,373 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/164cdd2be30e424692f1d6646d33c914 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/164cdd2be30e424692f1d6646d33c914 2023-04-28 14:58:07,376 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/164cdd2be30e424692f1d6646d33c914, entries=150, sequenceid=109, filesize=11.6 K 2023-04-28 14:58:07,376 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~391.75 KB/401152, currentSize=40.25 KB/41220 for f40347916069de574ae857f0751743f8 in 1250ms, sequenceid=109, compaction requested=true 2023-04-28 14:58:07,376 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:07,377 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:07,377 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:07,377 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:07,377 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:07,377 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:07,377 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:07,378 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:07,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:07,378 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:07,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:07,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:07,378 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:07,378 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:07,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:07,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:07,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:07,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:07,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:07,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:07,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6373f92c[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:07,379 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:07,379 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:07,379 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:07,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:07,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:07,379 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:07,379 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:07,379 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:07,380 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:07,380 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:07,380 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:07,380 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:07,380 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:07,380 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:07,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:07,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:07,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:07,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:07,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:07,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:07,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:07,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:07,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:07,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:07,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:07,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:07,382 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:07,382 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:07,382 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:07,382 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:07,382 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:07,384 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:07,384 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:07,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:07,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:07,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:07,384 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:07,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:07,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:07,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:07,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:07,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:07,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:07,384 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:07,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:07,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:07,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:07,386 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:07,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:07,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:07,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:07,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:07,387 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:07,387 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:07,387 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:07,387 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:07,387 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:07,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:07,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:07,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:07,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:07,387 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:07,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:07,387 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:07,387 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:07,388 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:07,388 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:07,388 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:07,388 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:07,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:07,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:07,488 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:07,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:07,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:07,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:07,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:07,491 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:07,491 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:07,491 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:07,491 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:07,491 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:07,491 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:07,492 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:07,492 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:07,492 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:07,492 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:07,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:07,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:07,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:07,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:07,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:07,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:07,493 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:07,498 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:07,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:07,498 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:07,498 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:07,498 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:07,498 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:07,498 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:07,498 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:07,498 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:07,498 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f40347916069de574ae857f0751743f8 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-28 14:58:07,499 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=A 2023-04-28 14:58:07,499 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:07,499 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=B 2023-04-28 14:58:07,499 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:07,499 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=C 2023-04-28 14:58:07,499 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:07,905 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/ac2ecd90132941448718a1603cff8c42 2023-04-28 14:58:07,915 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/96a1730e0f8949828aee839bde99ea15 2023-04-28 14:58:08,327 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/c89dcbf29ed242cc912d4a3763ea18f8 2023-04-28 14:58:08,330 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/ac2ecd90132941448718a1603cff8c42 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/ac2ecd90132941448718a1603cff8c42 2023-04-28 14:58:08,334 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/ac2ecd90132941448718a1603cff8c42, entries=150, sequenceid=118, filesize=11.6 K 2023-04-28 14:58:08,334 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/96a1730e0f8949828aee839bde99ea15 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/96a1730e0f8949828aee839bde99ea15 2023-04-28 14:58:08,338 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/96a1730e0f8949828aee839bde99ea15, entries=150, sequenceid=118, filesize=11.6 K 2023-04-28 14:58:08,338 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/c89dcbf29ed242cc912d4a3763ea18f8 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/c89dcbf29ed242cc912d4a3763ea18f8 2023-04-28 14:58:08,342 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/c89dcbf29ed242cc912d4a3763ea18f8, entries=150, sequenceid=118, filesize=11.6 K 2023-04-28 14:58:08,343 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for f40347916069de574ae857f0751743f8 in 845ms, sequenceid=118, compaction requested=true 2023-04-28 14:58:08,343 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:08,343 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:08,343 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:08,343 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:08,343 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,343 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:08,343 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:08,344 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,344 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:08,344 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,344 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:08,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:08,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:08,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@138873f7[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:08,345 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:08,345 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,346 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,346 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:08,346 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:08,346 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:08,346 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,346 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:08,348 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,348 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:08,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:08,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:08,348 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:08,348 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:08,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:08,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:08,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:08,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:08,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:08,351 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:08,351 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:08,351 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:08,351 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:08,351 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:08,352 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,352 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,352 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:08,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:08,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:08,352 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:08,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:08,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:08,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:08,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,354 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on f40347916069de574ae857f0751743f8 2023-04-28 14:58:08,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:08,354 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f40347916069de574ae857f0751743f8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 14:58:08,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,361 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=A 2023-04-28 14:58:08,361 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:08,361 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=B 2023-04-28 14:58:08,361 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:08,361 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:08,361 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:08,361 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:08,361 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:08,361 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:08,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:08,362 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=C 2023-04-28 14:58:08,362 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:08,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:08,362 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:08,362 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:08,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:08,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:08,362 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:08,362 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,362 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:08,362 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,362 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:08,369 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/d2627b4b297c481b8459a87a58bbfd99 2023-04-28 14:58:08,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682693888371, max timestamp=1682693888380Num uniques 859; 2023-04-28 14:58:08,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 14:58:08,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682693888371, max timestamp=1682693888380Num uniques 859; 2023-04-28 14:58:08,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 14:58:08,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682693888371, max timestamp=1682693888380Num uniques 859; 2023-04-28 14:58:08,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 14:58:08,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:08,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693948392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:08,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693948397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:08,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693948397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:08,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693948397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,401 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:08,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50270 deadline: 1682693948398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,462 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:08,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:08,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:08,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:08,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:08,465 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:08,465 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:08,465 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:08,465 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:08,466 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,466 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:08,467 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:08,467 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:08,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:08,467 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,467 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:08,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:08,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:08,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:08,467 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:08,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:08,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:08,468 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:08,468 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:08,468 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:08,469 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:08,469 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:08,470 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:08,470 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:08,470 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:08,470 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:08,470 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:08,470 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:08,470 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,470 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:08,470 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:08,471 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:08,471 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:08,472 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,472 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:08,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:08,472 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:08,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d7bc872[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:08,472 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,473 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,473 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:08,473 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,473 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:08,473 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:08,473 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:08,474 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,474 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:08,474 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:08,474 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:08,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:08,476 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:08,476 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:08,476 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:08,476 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:08,476 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:08,478 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:08,478 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,479 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:08,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:08,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:08,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:08,484 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:08,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,486 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:08,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,489 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,489 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:08,489 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:08,489 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:08,490 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:08,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:08,490 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,490 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,490 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:08,490 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:08,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:08,490 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:08,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:08,490 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:08,490 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:08,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:08,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:08,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:08,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693948498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:08,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693948500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:08,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693948501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:08,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693948501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:08,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50270 deadline: 1682693948501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,590 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:08,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:08,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:08,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:08,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:08,593 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:08,593 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:08,593 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:08,594 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:08,594 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,594 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:08,594 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:08,594 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:08,595 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,595 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:08,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:08,595 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:08,595 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:08,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:08,596 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:08,596 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:08,596 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:08,596 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:08,596 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:08,596 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:08,596 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:08,596 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:08,596 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:08,596 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:08,596 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:08,596 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,596 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:08,596 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:08,597 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,597 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:08,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:08,597 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,597 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:08,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:08,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:08,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:08,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:08,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38c95fec[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:08,598 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:08,598 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,599 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,599 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:08,599 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:08,599 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,599 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:08,599 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:08,600 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,600 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:08,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:08,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:08,600 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:08,600 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:08,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:08,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:08,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:08,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:08,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:08,602 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:08,602 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:08,602 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:08,602 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:08,602 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:08,603 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,603 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:08,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:08,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,603 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:08,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:08,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:08,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:08,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,604 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:08,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,605 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:08,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,606 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,606 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:08,606 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:08,606 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,606 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:08,606 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:08,606 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:08,606 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:08,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:08,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:08,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:08,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:08,606 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,606 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,606 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:08,606 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:08,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:08,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:08,705 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:08,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693948702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,707 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:08,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:08,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693948703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:08,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:08,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693948704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:08,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693948704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:08,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:08,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:08,709 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:08,709 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:08,709 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:08,709 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:08,710 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,710 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:08,710 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:08,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:08,710 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:08,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:08,711 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,711 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:08,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,711 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:08,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:08,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:08,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50270 deadline: 1682693948708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,711 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:08,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:08,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:08,712 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:08,712 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:08,712 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:08,712 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:08,712 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:08,712 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:08,712 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:08,712 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:08,712 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:08,712 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:08,712 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:08,712 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,712 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:08,712 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:08,713 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,713 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:08,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:08,713 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,713 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:08,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:08,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:08,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:08,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:08,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@448f7b40[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:08,714 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:08,714 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,715 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,715 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:08,715 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:08,715 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:08,715 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,715 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:08,716 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,716 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:08,716 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:08,716 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:08,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:08,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:08,717 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:08,717 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:08,717 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:08,717 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:08,717 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:08,719 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,719 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:08,719 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,719 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,720 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:08,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,723 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:08,723 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:08,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:08,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:08,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:08,723 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:08,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:08,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:08,723 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:08,723 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:08,726 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,726 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:08,726 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:08,726 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,726 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,726 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,726 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:08,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:08,781 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/c1ef58e2a8b84d7d81680ba5e49448f6 2023-04-28 14:58:08,823 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:08,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:08,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:08,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:08,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:08,827 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:08,827 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:08,827 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:08,827 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:08,827 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,827 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:08,828 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,828 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:08,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:08,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:08,828 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,828 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:08,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:08,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:08,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:08,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:08,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:08,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:08,830 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:08,830 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:08,830 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:08,830 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:08,830 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:08,830 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:08,830 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:08,830 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:08,830 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:08,830 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:08,830 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:08,830 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,830 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:08,830 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:08,832 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,832 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:08,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:08,832 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:08,832 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:08,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:08,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:08,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:08,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14c8bfed[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:08,834 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:08,834 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,834 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,834 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:08,834 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:08,834 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,834 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:08,834 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:08,835 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,835 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:08,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:08,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:08,835 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:08,835 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:08,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:08,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:08,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:08,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:08,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:08,838 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:08,838 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:08,838 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:08,838 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:08,838 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:08,839 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,839 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,839 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:08,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,839 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:08,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:08,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,841 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:08,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,843 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,843 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:08,843 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:08,843 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,843 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:08,844 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:08,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:08,844 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:08,843 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,844 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,844 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:08,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:08,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:08,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:08,844 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:08,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:08,844 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:08,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:08,944 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:08,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:08,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:08,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:08,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:08,947 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:08,947 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:08,947 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:08,947 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:08,948 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,948 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:08,948 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:08,948 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:08,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:08,949 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,949 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:08,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:08,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:08,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:08,950 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:08,950 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:08,950 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:08,950 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:08,951 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:08,951 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:08,951 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:08,951 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:08,951 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:08,951 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:08,951 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:08,951 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:08,951 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:08,951 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:08,951 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,951 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:08,951 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:08,952 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,952 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:08,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:08,952 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,952 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:08,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:08,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:08,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:08,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:08,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@763cd5e[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:08,953 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:08,953 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,954 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,954 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:08,954 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:08,954 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:08,954 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,954 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:08,955 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:08,955 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:08,955 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:08,955 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:08,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:08,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:08,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:08,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:08,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:08,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:08,958 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:08,958 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:08,958 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:08,958 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:08,958 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:08,959 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,959 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,959 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:08,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:08,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:08,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:08,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:08,959 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:08,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:08,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,960 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:08,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:08,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,962 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,962 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:08,962 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:08,962 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:08,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:08,962 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:08,962 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:08,962 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:08,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:08,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:08,962 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:08,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:08,962 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:08,963 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:08,963 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:08,963 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:08,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:08,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:09,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:09,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693949006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,012 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:09,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:09,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693949008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693949009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,012 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:09,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693949010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:09,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50270 deadline: 1682693949014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,063 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:09,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:09,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:09,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:09,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:09,066 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:09,066 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:09,066 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:09,066 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:09,066 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:09,066 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:09,067 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:09,067 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:09,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:09,067 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:09,067 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:09,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:09,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:09,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:09,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:09,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:09,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:09,068 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:09,068 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:09,068 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:09,068 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:09,068 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:09,068 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:09,069 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:09,069 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:09,069 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:09,069 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:09,069 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:09,069 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,069 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:09,069 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:09,070 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,070 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:09,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:09,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:09,070 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:09,070 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:09,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:09,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:09,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:09,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:09,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:09,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7527c461[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:09,072 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:09,072 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:09,072 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:09,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:09,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:09,073 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:09,073 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:09,073 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:09,073 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,073 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:09,073 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,073 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:09,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:09,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:09,074 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:09,074 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:09,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:09,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:09,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:09,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:09,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:09,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:09,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:09,077 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:09,077 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:09,077 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:09,077 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:09,077 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:09,078 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:09,078 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:09,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:09,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:09,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:09,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:09,078 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:09,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:09,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:09,079 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:09,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:09,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:09,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:09,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:09,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:09,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,080 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:09,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:09,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,082 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,082 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:09,082 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:09,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:09,082 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:09,082 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:09,082 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:09,082 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:09,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:09,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:09,082 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:09,082 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:09,083 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:09,083 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:09,083 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:09,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:09,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:09,183 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:09,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:09,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:09,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:09,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:09,186 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:09,186 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:09,186 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:09,186 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:09,186 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:09,186 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:09,187 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,187 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:09,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:09,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:09,187 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,187 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:09,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:09,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:09,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:09,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:09,188 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:09,188 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:09,188 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:09,188 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:09,188 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:09,188 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:09,188 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:09,188 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:09,188 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:09,188 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:09,188 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:09,188 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:09,188 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:09,188 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:09,188 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,188 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:09,188 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:09,189 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,189 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:09,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:09,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:09,189 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:09,189 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:09,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:09,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:09,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:09,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:09,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:09,191 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:09,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4acde4fe[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:09,191 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:09,191 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:09,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:09,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:09,191 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:09,191 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:09,191 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:09,191 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,191 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:09,192 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,192 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:09,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:09,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:09,192 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:09,192 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:09,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:09,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:09,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:09,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:09,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:09,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:09,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:09,194 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:09,194 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:09,194 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:09,194 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:09,194 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:09,195 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:09,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:09,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:09,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:09,195 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:09,195 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:09,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:09,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:09,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:09,195 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:09,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:09,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:09,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:09,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:09,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:09,196 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:09,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:09,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,197 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:09,197 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:09,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:09,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:09,198 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,198 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:09,198 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:09,198 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,198 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:09,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:09,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:09,198 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:09,198 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:09,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:09,198 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:09,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:09,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:09,198 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:09,198 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:09,202 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/fd4c090f866242f0a419540aa3de8e56 2023-04-28 14:58:09,206 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/d2627b4b297c481b8459a87a58bbfd99 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/d2627b4b297c481b8459a87a58bbfd99 2023-04-28 14:58:09,209 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/d2627b4b297c481b8459a87a58bbfd99, entries=150, sequenceid=129, filesize=11.7 K 2023-04-28 14:58:09,210 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/c1ef58e2a8b84d7d81680ba5e49448f6 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/c1ef58e2a8b84d7d81680ba5e49448f6 2023-04-28 14:58:09,214 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/c1ef58e2a8b84d7d81680ba5e49448f6, entries=150, sequenceid=129, filesize=11.7 K 2023-04-28 14:58:09,214 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/fd4c090f866242f0a419540aa3de8e56 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/fd4c090f866242f0a419540aa3de8e56 2023-04-28 14:58:09,217 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/fd4c090f866242f0a419540aa3de8e56, entries=150, sequenceid=129, filesize=11.7 K 2023-04-28 14:58:09,218 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for f40347916069de574ae857f0751743f8 in 864ms, sequenceid=129, compaction requested=true 2023-04-28 14:58:09,218 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:09,218 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:09,218 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 14:58:09,218 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 14:58:09,218 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:09,219 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:09,220 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57274 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 14:58:09,220 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): f40347916069de574ae857f0751743f8/B is initiating minor compaction (all files) 2023-04-28 14:58:09,220 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f40347916069de574ae857f0751743f8/B in TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:09,220 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/7f2dd54a6d87480383080f0131188611, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/f6110f1425ea4ac78a05b41dd3e2137a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/97ac3ad5d5fc4bd283fd4ef5a5c3200e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/96a1730e0f8949828aee839bde99ea15, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/c1ef58e2a8b84d7d81680ba5e49448f6] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp, totalSize=55.9 K 2023-04-28 14:58:09,220 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57274 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 14:58:09,220 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): f40347916069de574ae857f0751743f8/A is initiating minor compaction (all files) 2023-04-28 14:58:09,220 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f40347916069de574ae857f0751743f8/A in TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:09,221 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 7f2dd54a6d87480383080f0131188611, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682693884869 2023-04-28 14:58:09,221 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/94d8d51e11534d59bf433357f9f830f1, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/64aa34bfcbea4485a9f771c9860e6cef, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/63bc9271a4ee4fe286cf9f7cefef0483, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/ac2ecd90132941448718a1603cff8c42, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/d2627b4b297c481b8459a87a58bbfd99] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp, totalSize=55.9 K 2023-04-28 14:58:09,221 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting f6110f1425ea4ac78a05b41dd3e2137a, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1682693884988 2023-04-28 14:58:09,221 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 94d8d51e11534d59bf433357f9f830f1, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682693884869 2023-04-28 14:58:09,221 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 97ac3ad5d5fc4bd283fd4ef5a5c3200e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1682693885618 2023-04-28 14:58:09,221 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 64aa34bfcbea4485a9f771c9860e6cef, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1682693884988 2023-04-28 14:58:09,221 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 96a1730e0f8949828aee839bde99ea15, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1682693886231 2023-04-28 14:58:09,222 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 63bc9271a4ee4fe286cf9f7cefef0483, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1682693885618 2023-04-28 14:58:09,222 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting c1ef58e2a8b84d7d81680ba5e49448f6, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1682693888352 2023-04-28 14:58:09,223 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting ac2ecd90132941448718a1603cff8c42, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1682693886231 2023-04-28 14:58:09,223 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting d2627b4b297c481b8459a87a58bbfd99, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1682693888352 2023-04-28 14:58:09,230 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): f40347916069de574ae857f0751743f8#B#compaction#207 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:09,231 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): f40347916069de574ae857f0751743f8#A#compaction#208 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:09,259 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/6ace4e44c3ce479aaa2975c7d498f653 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/6ace4e44c3ce479aaa2975c7d498f653 2023-04-28 14:58:09,264 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in f40347916069de574ae857f0751743f8/B of f40347916069de574ae857f0751743f8 into 6ace4e44c3ce479aaa2975c7d498f653(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:09,264 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:09,264 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8., storeName=f40347916069de574ae857f0751743f8/B, priority=11, startTime=1682693889218; duration=0sec 2023-04-28 14:58:09,265 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:09,265 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 14:58:09,266 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57274 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 14:58:09,267 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): f40347916069de574ae857f0751743f8/C is initiating minor compaction (all files) 2023-04-28 14:58:09,267 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f40347916069de574ae857f0751743f8/C in TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:09,267 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/76a6167544fa43fcb34e87cd498a36c3, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/fcb2840355cd4db49c8d5ea1461284fa, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/164cdd2be30e424692f1d6646d33c914, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/c89dcbf29ed242cc912d4a3763ea18f8, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/fd4c090f866242f0a419540aa3de8e56] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp, totalSize=55.9 K 2023-04-28 14:58:09,267 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 76a6167544fa43fcb34e87cd498a36c3, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682693884869 2023-04-28 14:58:09,268 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting fcb2840355cd4db49c8d5ea1461284fa, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1682693884988 2023-04-28 14:58:09,268 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 164cdd2be30e424692f1d6646d33c914, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1682693885618 2023-04-28 14:58:09,268 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting c89dcbf29ed242cc912d4a3763ea18f8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1682693886231 2023-04-28 14:58:09,268 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting fd4c090f866242f0a419540aa3de8e56, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1682693888352 2023-04-28 14:58:09,280 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): f40347916069de574ae857f0751743f8#C#compaction#209 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:09,298 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:09,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:09,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:09,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:09,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:09,301 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:09,301 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:09,301 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:09,301 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:09,302 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:09,302 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:09,302 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,302 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:09,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:09,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:09,303 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,303 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:09,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:09,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:09,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:09,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:09,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:09,303 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:09,303 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:09,304 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:09,304 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:09,304 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:09,304 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:09,304 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:09,304 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:09,304 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:09,304 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f40347916069de574ae857f0751743f8 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-28 14:58:09,304 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=A 2023-04-28 14:58:09,304 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:09,304 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=B 2023-04-28 14:58:09,304 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:09,304 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=C 2023-04-28 14:58:09,305 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:09,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on f40347916069de574ae857f0751743f8 2023-04-28 14:58:09,511 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:09,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:09,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50270 deadline: 1682693949518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,522 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:09,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693949518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,522 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:09,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693949519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:09,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693949520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:09,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693949520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,624 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:09,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693949623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:09,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693949623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,626 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:09,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693949625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,627 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:09,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693949625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,647 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/7a27d40375534ea5aba4d349aac90c32 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/7a27d40375534ea5aba4d349aac90c32 2023-04-28 14:58:09,651 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in f40347916069de574ae857f0751743f8/A of f40347916069de574ae857f0751743f8 into 7a27d40375534ea5aba4d349aac90c32(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:09,651 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:09,651 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8., storeName=f40347916069de574ae857f0751743f8/A, priority=11, startTime=1682693889218; duration=0sec 2023-04-28 14:58:09,651 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:09,697 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/7668bf346c1a436c908bae1e232fa3b0 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/7668bf346c1a436c908bae1e232fa3b0 2023-04-28 14:58:09,701 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in f40347916069de574ae857f0751743f8/C of f40347916069de574ae857f0751743f8 into 7668bf346c1a436c908bae1e232fa3b0(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:09,701 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:09,701 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8., storeName=f40347916069de574ae857f0751743f8/C, priority=11, startTime=1682693889219; duration=0sec 2023-04-28 14:58:09,702 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:09,716 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/e2098ac32c3c4aabb78f456a864e2418 2023-04-28 14:58:09,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:09,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693949835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,836 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:09,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693949835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,836 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:09,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693949835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:09,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:09,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693949835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,130 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/2c4fb06a03e74bdeb5e9b84db8a509b7 2023-04-28 14:58:10,139 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/a22526c414c5406886bae20ba16b4ade 2023-04-28 14:58:10,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:10,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693950138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,140 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:10,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693950138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:10,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693950139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:10,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693950139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,143 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/e2098ac32c3c4aabb78f456a864e2418 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/e2098ac32c3c4aabb78f456a864e2418 2023-04-28 14:58:10,148 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/e2098ac32c3c4aabb78f456a864e2418, entries=150, sequenceid=165, filesize=11.8 K 2023-04-28 14:58:10,149 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/2c4fb06a03e74bdeb5e9b84db8a509b7 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/2c4fb06a03e74bdeb5e9b84db8a509b7 2023-04-28 14:58:10,153 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/2c4fb06a03e74bdeb5e9b84db8a509b7, entries=150, sequenceid=165, filesize=11.8 K 2023-04-28 14:58:10,154 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/a22526c414c5406886bae20ba16b4ade as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/a22526c414c5406886bae20ba16b4ade 2023-04-28 14:58:10,157 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/a22526c414c5406886bae20ba16b4ade, entries=150, sequenceid=165, filesize=11.8 K 2023-04-28 14:58:10,158 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for f40347916069de574ae857f0751743f8 in 854ms, sequenceid=165, compaction requested=false 2023-04-28 14:58:10,158 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:10,158 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:10,158 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:10,158 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:10,158 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,158 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:10,158 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:10,159 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,159 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:10,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:10,159 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,159 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:10,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:10,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:10,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:10,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:10,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:10,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b2b1d9d[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:10,161 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:10,162 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,162 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,162 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,162 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:10,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,163 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:10,163 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:10,163 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:10,163 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:10,163 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:10,163 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:10,163 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:10,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:10,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:10,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:10,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:10,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:10,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:10,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:10,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:10,166 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:10,166 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:10,166 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:10,166 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:10,166 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:10,167 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,167 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:10,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:10,167 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:10,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:10,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:10,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:10,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,168 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:10,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:10,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:10,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,169 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:10,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:10,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,170 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,171 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:10,171 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:10,171 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,171 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,171 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,171 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:10,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:10,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:10,171 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:10,171 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:10,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:10,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:10,171 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,171 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:10,171 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:10,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:10,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:10,271 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:10,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:10,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:10,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:10,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:10,274 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:10,274 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:10,274 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:10,274 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:10,274 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,274 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:10,275 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:10,275 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:10,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:10,275 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,275 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:10,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:10,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:10,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:10,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:10,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:10,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:10,276 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:10,276 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:10,276 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:10,276 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:10,276 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:10,276 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:10,276 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:10,276 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f40347916069de574ae857f0751743f8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 14:58:10,277 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=A 2023-04-28 14:58:10,277 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:10,277 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=B 2023-04-28 14:58:10,277 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:10,277 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=C 2023-04-28 14:58:10,277 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:10,283 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=177 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/15876e573f484482bc1ae855b967981b 2023-04-28 14:58:10,299 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=177 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/491ac303e7b540ae85ad61232bfd36dd 2023-04-28 14:58:10,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on f40347916069de574ae857f0751743f8 2023-04-28 14:58:10,524 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:10,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693890552, max timestamp=1682693890566Num uniques 859; 2023-04-28 14:58:10,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:58:10,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693890552, max timestamp=1682693890566Num uniques 859; 2023-04-28 14:58:10,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:58:10,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693890552, max timestamp=1682693890566Num uniques 859; 2023-04-28 14:58:10,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:58:10,624 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:10,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50270 deadline: 1682693950622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,641 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:10,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693950640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:10,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693950640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:10,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693950641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,646 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:10,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693950645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,709 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=177 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/afc207f45af141c7b714ac7f60d1e8e1 2023-04-28 14:58:10,712 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/15876e573f484482bc1ae855b967981b as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/15876e573f484482bc1ae855b967981b 2023-04-28 14:58:10,715 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/15876e573f484482bc1ae855b967981b, entries=150, sequenceid=177, filesize=11.8 K 2023-04-28 14:58:10,715 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/491ac303e7b540ae85ad61232bfd36dd as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/491ac303e7b540ae85ad61232bfd36dd 2023-04-28 14:58:10,718 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/491ac303e7b540ae85ad61232bfd36dd, entries=150, sequenceid=177, filesize=11.8 K 2023-04-28 14:58:10,719 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/afc207f45af141c7b714ac7f60d1e8e1 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/afc207f45af141c7b714ac7f60d1e8e1 2023-04-28 14:58:10,723 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/afc207f45af141c7b714ac7f60d1e8e1, entries=150, sequenceid=177, filesize=11.8 K 2023-04-28 14:58:10,724 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for f40347916069de574ae857f0751743f8 in 448ms, sequenceid=177, compaction requested=true 2023-04-28 14:58:10,724 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:10,724 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:10,724 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:10,724 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:10,724 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,724 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:10,724 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:10,725 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,725 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:10,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:10,725 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,725 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:10,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:10,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:10,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:10,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:10,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682693890613, max timestamp=1682693890725Num uniques 1018; 2023-04-28 14:58:10,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682693890613, max timestamp=1682693890725Num uniques 1018; 2023-04-28 14:58:10,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-28 14:58:10,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:10,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-28 14:58:10,728 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:10,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on f40347916069de574ae857f0751743f8 2023-04-28 14:58:10,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23b2a9b[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:10,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682693890613, max timestamp=1682693890725Num uniques 1018; 2023-04-28 14:58:10,729 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f40347916069de574ae857f0751743f8 3/3 column families, dataSize=64.85 KB heapSize=178.80 KB 2023-04-28 14:58:10,728 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-28 14:58:10,729 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=A 2023-04-28 14:58:10,729 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:10,729 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=B 2023-04-28 14:58:10,729 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:10,729 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=C 2023-04-28 14:58:10,729 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:10,729 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,730 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:10,730 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:10,730 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:10,730 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,730 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:10,730 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:10,730 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,730 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:10,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:10,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:10,730 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:10,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:10,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:10,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:10,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:10,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:10,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:10,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:10,734 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:10,734 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:10,734 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:10,734 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:10,734 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:10,735 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:10,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:10,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:10,736 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,736 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:10,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,736 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/651de671b37e448db714e54c51162f6d 2023-04-28 14:58:10,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:10,736 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:10,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:10,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:10,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:10,738 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,740 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,740 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:10,740 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:10,740 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:10,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:10,740 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:10,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:10,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:10,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:10,740 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:10,741 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,741 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,741 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,741 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,741 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:10,741 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:10,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:10,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:10,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:10,748 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/24863ed37691480db93ce435d978d72b 2023-04-28 14:58:10,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693890745, max timestamp=1682693890772Num uniques 859; 2023-04-28 14:58:10,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693890745, max timestamp=1682693890772Num uniques 859; 2023-04-28 14:58:10,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:58:10,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693890745, max timestamp=1682693890772Num uniques 859; 2023-04-28 14:58:10,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:58:10,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:58:10,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682693890808, max timestamp=1682693890822Num uniques 1018; 2023-04-28 14:58:10,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-28 14:58:10,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682693890808, max timestamp=1682693890822Num uniques 1018; 2023-04-28 14:58:10,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-28 14:58:10,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682693890808, max timestamp=1682693890822Num uniques 1018; 2023-04-28 14:58:10,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-28 14:58:10,841 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:10,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:10,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:10,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:10,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:10,843 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:10,844 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:10,844 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:10,844 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:10,844 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,844 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:10,844 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,844 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:10,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:10,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:10,845 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,845 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:10,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:10,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:10,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:10,846 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:10,846 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:10,846 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:10,846 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:10,846 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:10,846 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:10,847 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:10,847 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:10,847 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:10,847 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:10,847 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:10,847 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:10,847 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:10,847 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:10,847 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,847 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:10,847 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:10,848 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,848 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:10,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:10,848 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,848 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:10,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:10,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:10,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:10,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:10,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:10,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4974e271[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:10,849 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:10,849 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,850 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,850 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:10,850 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:10,850 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:10,850 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,850 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:10,851 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,851 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:10,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:10,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:10,851 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:10,851 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:10,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:10,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:10,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:10,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:10,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:10,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:10,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:10,852 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:10,853 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:10,853 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:10,853 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:10,853 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:10,854 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,854 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:10,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:10,854 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:10,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:10,854 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:10,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:10,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:10,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:10,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:10,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:10,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:10,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,859 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,860 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,861 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:10,861 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:10,861 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:10,861 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:10,861 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:10,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:10,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:10,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:10,861 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:10,861 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,861 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:10,861 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,861 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:10,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:10,861 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:10,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682693890878, max timestamp=1682693890895Num uniques 1027; 2023-04-28 14:58:10,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682693890878, max timestamp=1682693890895Num uniques 1027; 2023-04-28 14:58:10,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-28 14:58:10,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-28 14:58:10,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682693890878, max timestamp=1682693890895Num uniques 1027; 2023-04-28 14:58:10,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-28 14:58:10,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682693890933, max timestamp=1682693890946Num uniques 1036; 2023-04-28 14:58:10,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682693890933, max timestamp=1682693890946Num uniques 1036; 2023-04-28 14:58:10,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-28 14:58:10,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682693890933, max timestamp=1682693890946Num uniques 1036; 2023-04-28 14:58:10,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-28 14:58:10,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-28 14:58:10,961 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:10,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:10,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:10,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:10,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:10,964 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:10,964 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:10,964 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:10,964 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:10,964 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,964 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:10,965 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:10,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:10,965 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:10,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:10,965 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,965 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:10,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:10,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:10,966 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:10,966 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:10,966 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:10,966 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:10,966 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:10,966 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:10,966 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:10,966 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:10,966 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:10,966 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:10,966 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:10,966 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:10,966 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:10,966 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:10,967 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,967 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:10,967 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:10,970 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,970 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:10,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:10,970 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,970 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:10,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:10,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:10,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:10,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:10,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:10,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c3b15b5[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:10,972 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:10,972 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,972 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,972 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:10,972 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:10,972 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:10,973 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,973 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:10,973 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:10,973 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,973 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:10,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:10,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:10,974 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:10,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:10,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:10,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:10,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:10,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:10,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:10,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:10,976 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:10,976 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:10,976 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:10,976 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:10,976 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:10,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:10,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:10,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:10,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:10,977 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:10,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:10,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:10,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:10,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:10,979 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:10,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:10,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,981 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:10,981 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:10,981 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,981 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:10,981 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:10,981 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:10,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:10,981 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,981 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:10,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:10,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:10,981 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:10,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:10,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:10,982 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:10,982 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:10,982 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:10,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:10,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:11,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682693890971, max timestamp=1682693891000Num uniques 1045; 2023-04-28 14:58:11,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-28 14:58:11,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682693890971, max timestamp=1682693891000Num uniques 1045; 2023-04-28 14:58:11,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-28 14:58:11,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682693890971, max timestamp=1682693891000Num uniques 1045; 2023-04-28 14:58:11,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-28 14:58:11,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682693891034, max timestamp=1682693891056Num uniques 1054; 2023-04-28 14:58:11,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-28 14:58:11,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682693891034, max timestamp=1682693891056Num uniques 1054; 2023-04-28 14:58:11,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682693891034, max timestamp=1682693891056Num uniques 1054; 2023-04-28 14:58:11,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-28 14:58:11,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-28 14:58:11,086 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:11,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:11,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:11,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:11,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:11,089 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:11,089 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:11,089 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:11,089 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:11,090 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,090 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,090 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:11,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:11,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:11,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,091 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:11,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:11,092 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,092 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:11,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:11,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:11,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:11,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:11,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:11,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:11,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:11,093 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:11,093 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:11,093 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:11,093 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:11,093 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:11,093 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:11,093 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,093 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:11,093 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:11,094 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,094 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:11,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:11,094 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,094 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:11,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:11,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:11,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:11,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:11,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fbf3a5a[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:11,096 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:11,096 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,096 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,096 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:11,096 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:11,096 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:11,098 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:11,098 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:11,098 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:11,098 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,098 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:11,098 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:11,098 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:11,098 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:11,098 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:11,098 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:11,099 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:11,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:11,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:11,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:11,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:11,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:11,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:11,100 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:11,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:11,102 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,104 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,104 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,104 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,104 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,104 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,104 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,104 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,104 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:11,104 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:11,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:11,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:11,104 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:11,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:11,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:11,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:11,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:11,106 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:11,106 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:11,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682693891085, max timestamp=1682693891119Num uniques 1013; 2023-04-28 14:58:11,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-28 14:58:11,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682693891085, max timestamp=1682693891119Num uniques 1013; 2023-04-28 14:58:11,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-28 14:58:11,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682693891085, max timestamp=1682693891119Num uniques 1013; 2023-04-28 14:58:11,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-28 14:58:11,156 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/3b4c92935cfa43dda9256e561158472a 2023-04-28 14:58:11,161 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/651de671b37e448db714e54c51162f6d as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/651de671b37e448db714e54c51162f6d 2023-04-28 14:58:11,166 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/651de671b37e448db714e54c51162f6d, entries=200, sequenceid=215, filesize=14.1 K 2023-04-28 14:58:11,167 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/24863ed37691480db93ce435d978d72b as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/24863ed37691480db93ce435d978d72b 2023-04-28 14:58:11,171 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/24863ed37691480db93ce435d978d72b, entries=150, sequenceid=215, filesize=11.8 K 2023-04-28 14:58:11,172 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/3b4c92935cfa43dda9256e561158472a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/3b4c92935cfa43dda9256e561158472a 2023-04-28 14:58:11,175 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/3b4c92935cfa43dda9256e561158472a, entries=150, sequenceid=215, filesize=11.8 K 2023-04-28 14:58:11,175 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~65.34 KB/66912, currentSize=114.05 KB/116790 for f40347916069de574ae857f0751743f8 in 447ms, sequenceid=215, compaction requested=true 2023-04-28 14:58:11,176 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:11,176 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:11,176 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:58:11,176 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:58:11,176 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:11,176 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:11,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on f40347916069de574ae857f0751743f8 2023-04-28 14:58:11,177 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f40347916069de574ae857f0751743f8 3/3 column families, dataSize=120.76 KB heapSize=344.77 KB 2023-04-28 14:58:11,177 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=A 2023-04-28 14:58:11,177 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:11,177 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=B 2023-04-28 14:58:11,177 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:11,177 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=C 2023-04-28 14:58:11,177 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:11,179 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48352 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:58:11,180 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): f40347916069de574ae857f0751743f8/B is initiating minor compaction (all files) 2023-04-28 14:58:11,180 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f40347916069de574ae857f0751743f8/B in TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:11,180 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/6ace4e44c3ce479aaa2975c7d498f653, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/2c4fb06a03e74bdeb5e9b84db8a509b7, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/491ac303e7b540ae85ad61232bfd36dd, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/24863ed37691480db93ce435d978d72b] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp, totalSize=47.2 K 2023-04-28 14:58:11,180 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50742 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:58:11,180 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): f40347916069de574ae857f0751743f8/A is initiating minor compaction (all files) 2023-04-28 14:58:11,180 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f40347916069de574ae857f0751743f8/A in TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:11,180 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/7a27d40375534ea5aba4d349aac90c32, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/e2098ac32c3c4aabb78f456a864e2418, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/15876e573f484482bc1ae855b967981b, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/651de671b37e448db714e54c51162f6d] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp, totalSize=49.6 K 2023-04-28 14:58:11,182 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 7a27d40375534ea5aba4d349aac90c32, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1682693888352 2023-04-28 14:58:11,182 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 6ace4e44c3ce479aaa2975c7d498f653, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1682693888352 2023-04-28 14:58:11,182 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting e2098ac32c3c4aabb78f456a864e2418, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1682693888390 2023-04-28 14:58:11,182 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 2c4fb06a03e74bdeb5e9b84db8a509b7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1682693888390 2023-04-28 14:58:11,183 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 15876e573f484482bc1ae855b967981b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1682693889510 2023-04-28 14:58:11,183 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 491ac303e7b540ae85ad61232bfd36dd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1682693889510 2023-04-28 14:58:11,183 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 651de671b37e448db714e54c51162f6d, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1682693890613 2023-04-28 14:58:11,183 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 24863ed37691480db93ce435d978d72b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1682693890613 2023-04-28 14:58:11,208 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): f40347916069de574ae857f0751743f8#B#compaction#220 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:11,208 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:11,213 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:11,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50270 deadline: 1682693951211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,216 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): f40347916069de574ae857f0751743f8#A#compaction#221 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:11,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:11,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:11,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:11,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:11,222 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:11,222 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:11,222 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:11,222 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:11,223 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,223 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,226 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,226 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:11,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:11,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:11,226 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,226 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:11,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:11,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,227 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:11,231 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:11,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:11,231 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:11,231 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:11,231 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:11,235 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:11,235 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:11,235 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/e5c0df22792a413b8dda6341feaafd5a 2023-04-28 14:58:11,235 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:11,235 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:11,235 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:11,235 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:11,235 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:11,235 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:11,235 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,238 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:11,238 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:11,240 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:11,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:11,240 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:11,240 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,240 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:11,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:11,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:11,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:11,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@740bb2c4[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:11,242 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:11,242 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,242 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,242 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:11,242 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:11,242 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,243 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:11,242 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:11,244 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,244 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:11,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:11,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:11,244 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:11,244 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:11,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:11,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:11,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:11,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:11,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:11,246 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:11,246 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:11,246 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:11,246 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:11,246 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:11,247 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:11,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:11,247 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,247 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:11,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:11,247 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:11,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:11,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:11,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,248 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:11,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,249 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,249 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:11,249 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:11,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:11,249 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:11,249 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:11,249 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:11,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:11,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:11,249 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,249 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,250 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,250 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,249 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:11,250 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:11,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:11,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:11,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:11,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50270 deadline: 1682693951314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,350 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:11,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:11,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:11,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:11,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:11,353 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:11,353 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:11,353 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:11,353 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:11,353 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,353 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,354 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:11,354 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:11,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:11,354 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,354 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:11,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:11,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,355 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:11,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:11,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:11,355 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:11,355 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:11,356 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:11,356 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:11,356 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:11,356 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:11,356 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:11,356 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:11,356 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:11,356 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:11,356 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:11,356 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,356 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:11,356 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:11,357 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,357 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:11,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:11,358 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,358 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:11,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:11,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:11,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:11,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:11,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a82b15e[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:11,359 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:11,359 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,359 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,360 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:11,360 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:11,360 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:11,360 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,360 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:11,360 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:11,360 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,360 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:11,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:11,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:11,360 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:11,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:11,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:11,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:11,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:11,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:11,362 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:11,362 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:11,362 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:11,362 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:11,362 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:11,363 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,363 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:11,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:11,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,363 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:11,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:11,364 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:11,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:11,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:11,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,365 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:11,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,366 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,366 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,366 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,366 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,366 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,366 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:11,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:11,366 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:11,366 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,366 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:11,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:11,366 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:11,366 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:11,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:11,366 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:11,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:11,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:11,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:11,466 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:11,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:11,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:11,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:11,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:11,469 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:11,469 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:11,469 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:11,469 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:11,470 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,470 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,470 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,470 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:11,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:11,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:11,471 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,471 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:11,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:11,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,471 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:11,472 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:11,472 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:11,472 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:11,472 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:11,472 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:11,472 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:11,472 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:11,472 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:11,472 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:11,472 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:11,472 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:11,472 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:11,472 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:11,472 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,472 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:11,472 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:11,473 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,473 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:11,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:11,473 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,473 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:11,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:11,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:11,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:11,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:11,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e208906[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:11,475 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:11,475 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,475 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,475 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:11,475 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:11,475 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:11,475 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,475 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:11,476 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,476 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:11,476 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:11,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:11,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:11,476 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:11,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:11,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:11,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:11,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:11,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:11,479 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:11,479 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:11,479 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:11,479 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:11,479 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:11,480 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,480 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:11,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:11,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,480 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:11,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:11,480 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:11,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:11,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:11,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:11,481 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,489 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:11,489 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:11,489 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:11,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:11,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:11,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:11,489 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,489 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:11,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:11,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:11,489 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,489 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:11,490 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,490 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:11,490 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,490 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,490 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:11,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50270 deadline: 1682693951519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,590 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:11,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:11,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:11,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:11,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:11,593 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:11,593 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:11,593 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:11,593 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:11,593 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,593 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,594 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:11,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:11,594 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:11,594 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,594 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:11,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:11,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,595 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:11,595 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:11,595 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:11,595 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:11,595 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:11,595 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:11,595 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:11,595 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:11,595 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:11,595 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:11,595 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:11,595 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:11,595 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:11,595 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:11,596 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,596 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:11,596 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:11,597 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,597 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:11,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:11,597 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:11,597 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:11,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:11,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:11,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:11,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@781b1b72[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:11,598 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:11,598 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,599 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,600 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:11,600 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,600 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:11,600 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:11,600 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:11,601 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,601 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:11,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:11,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:11,601 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:11,601 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:11,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:11,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:11,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:11,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:11,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:11,603 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:11,603 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:11,603 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:11,603 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:11,603 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:11,604 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,604 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:11,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:11,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,604 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:11,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:11,604 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:11,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:11,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:11,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,605 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:11,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,607 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,607 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,607 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,607 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:11,607 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:11,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:11,607 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:11,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:11,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:11,607 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:11,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:11,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:11,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:11,607 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:11,607 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,607 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:11,607 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,607 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,640 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/5f3b4fd07a2a4f89bcc94c29c5858265 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/5f3b4fd07a2a4f89bcc94c29c5858265 2023-04-28 14:58:11,644 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f40347916069de574ae857f0751743f8/A of f40347916069de574ae857f0751743f8 into 5f3b4fd07a2a4f89bcc94c29c5858265(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:11,644 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:11,644 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8., storeName=f40347916069de574ae857f0751743f8/A, priority=12, startTime=1682693891176; duration=0sec 2023-04-28 14:58:11,644 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:11,644 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:58:11,645 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/0d38f92b82664883a5b0eaaaf66ccb0c 2023-04-28 14:58:11,646 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48352 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:58:11,647 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:11,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693951645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,647 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): f40347916069de574ae857f0751743f8/C is initiating minor compaction (all files) 2023-04-28 14:58:11,647 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f40347916069de574ae857f0751743f8/C in TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:11,647 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/7668bf346c1a436c908bae1e232fa3b0, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/a22526c414c5406886bae20ba16b4ade, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/afc207f45af141c7b714ac7f60d1e8e1, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/3b4c92935cfa43dda9256e561158472a] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp, totalSize=47.2 K 2023-04-28 14:58:11,649 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 7668bf346c1a436c908bae1e232fa3b0, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1682693888352 2023-04-28 14:58:11,650 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting a22526c414c5406886bae20ba16b4ade, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1682693888390 2023-04-28 14:58:11,650 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting afc207f45af141c7b714ac7f60d1e8e1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1682693889510 2023-04-28 14:58:11,651 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 3b4c92935cfa43dda9256e561158472a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1682693890613 2023-04-28 14:58:11,651 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/4ad2bb2074a2413e9325584276381b30 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/4ad2bb2074a2413e9325584276381b30 2023-04-28 14:58:11,652 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:11,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693951650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,652 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:11,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693951650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:11,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693951651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,657 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f40347916069de574ae857f0751743f8/B of f40347916069de574ae857f0751743f8 into 4ad2bb2074a2413e9325584276381b30(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:11,657 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:11,657 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8., storeName=f40347916069de574ae857f0751743f8/B, priority=12, startTime=1682693891176; duration=0sec 2023-04-28 14:58:11,657 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:11,660 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): f40347916069de574ae857f0751743f8#C#compaction#224 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:11,707 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:11,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:11,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:11,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:11,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:11,710 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:11,710 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:11,710 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:11,710 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:11,711 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,711 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,711 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:11,711 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:11,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:11,712 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,712 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:11,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:11,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:11,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:11,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:11,712 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:11,712 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:11,712 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:11,714 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:11,714 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:11,714 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:11,714 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:11,714 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:11,714 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:11,714 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:11,714 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:11,714 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,714 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:11,714 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:11,715 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,715 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:11,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:11,715 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,715 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:11,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:11,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:11,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:11,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:11,716 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:11,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51af0b27[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:11,716 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,717 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,717 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:11,717 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:11,717 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:11,717 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,717 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:11,718 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,718 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:11,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:11,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:11,718 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:11,718 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:11,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:11,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:11,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:11,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:11,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:11,719 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:11,720 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:11,720 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:11,720 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:11,720 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:11,721 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,721 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,721 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:11,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:11,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:11,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:11,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:11,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:11,722 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:11,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,723 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:11,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,724 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,724 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:11,724 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:11,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:11,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:11,724 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:11,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:11,724 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:11,724 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,724 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,725 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,725 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,725 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:11,725 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:11,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:11,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:11,725 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:11,824 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:11,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50270 deadline: 1682693951822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,825 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:11,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:11,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:11,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:11,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:11,828 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:11,828 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:11,828 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:11,828 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:11,828 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,828 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,829 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:11,829 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:11,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:11,829 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,829 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:11,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:11,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:11,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:11,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:11,830 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:11,830 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:11,830 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:11,830 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:11,830 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:11,830 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:11,830 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:11,830 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:11,830 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:11,830 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:11,831 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:11,831 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,831 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:11,831 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:11,831 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,831 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:11,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:11,832 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,832 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:11,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:11,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:11,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:11,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:11,833 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:11,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e28580d[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:11,833 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,833 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,833 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:11,833 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:11,833 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:11,834 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,834 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:11,834 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,834 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:11,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:11,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:11,834 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:11,834 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:11,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:11,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:11,836 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:11,836 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:11,836 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:11,836 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:11,836 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:11,837 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,837 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:11,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:11,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,837 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:11,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:11,837 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:11,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,838 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:11,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,840 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:11,840 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:11,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:11,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:11,840 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,840 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,840 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:11,840 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:11,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:11,840 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,840 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,840 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,840 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,840 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:11,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:11,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:11,840 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:11,941 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:11,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:11,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:11,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:11,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:11,944 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:11,944 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:11,944 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:11,944 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:11,946 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,946 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,946 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:11,946 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:11,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:11,947 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,947 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:11,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:11,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,947 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:11,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:11,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:11,951 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:11,951 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:11,951 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:11,951 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:11,951 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:11,952 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:11,952 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:11,952 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:11,952 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:11,952 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:11,952 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:11,952 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,952 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:11,952 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:11,953 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,953 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:11,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:11,954 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,954 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:11,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:11,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:11,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:11,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:11,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74ea8277[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:11,957 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:11,957 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,958 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,958 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:11,958 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:11,958 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:11,960 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:11,961 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:11,961 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:11,961 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,961 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:11,961 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:11,961 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:11,961 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:11,961 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:11,961 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:11,961 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:11,962 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,962 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:11,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,962 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:11,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,963 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:11,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,964 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:11,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:11,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,966 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:11,966 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:11,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:11,966 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:11,966 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,966 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,966 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:11,966 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:11,966 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,966 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:11,966 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:11,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:11,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:11,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:11,967 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:11,967 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:11,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:11,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:11,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:12,058 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/4bc46f16287842ac8ed387e9fce2fec1 2023-04-28 14:58:12,062 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/e5c0df22792a413b8dda6341feaafd5a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/e5c0df22792a413b8dda6341feaafd5a 2023-04-28 14:58:12,065 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/e5c0df22792a413b8dda6341feaafd5a, entries=200, sequenceid=353, filesize=14.3 K 2023-04-28 14:58:12,065 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/0d38f92b82664883a5b0eaaaf66ccb0c as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/0d38f92b82664883a5b0eaaaf66ccb0c 2023-04-28 14:58:12,067 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:12,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:12,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:12,069 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/0d38f92b82664883a5b0eaaaf66ccb0c, entries=150, sequenceid=353, filesize=11.9 K 2023-04-28 14:58:12,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:12,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:12,070 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/4bc46f16287842ac8ed387e9fce2fec1 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/4bc46f16287842ac8ed387e9fce2fec1 2023-04-28 14:58:12,070 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:12,070 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:12,070 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:12,070 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:12,070 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:12,070 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:12,071 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:12,071 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:12,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:12,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:12,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:12,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:12,072 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:12,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:12,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:12,072 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:12,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:12,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:12,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:12,079 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/5b6c5c9671554807b0b64c75604e9e78 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/5b6c5c9671554807b0b64c75604e9e78 2023-04-28 14:58:12,082 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:12,082 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:12,082 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:12,083 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:12,083 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:12,083 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:12,084 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:12,084 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:12,084 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:12,084 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:12,084 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:12,084 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:12,084 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:12,084 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:12,084 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f40347916069de574ae857f0751743f8/C of f40347916069de574ae857f0751743f8 into 5b6c5c9671554807b0b64c75604e9e78(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:12,085 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:12,085 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8., storeName=f40347916069de574ae857f0751743f8/C, priority=12, startTime=1682693891176; duration=0sec 2023-04-28 14:58:12,085 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:12,085 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:12,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:12,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:12,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:12,086 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:12,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:12,086 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:12,086 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:12,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:12,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:12,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:12,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@540ad713[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:12,088 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:12,088 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:12,088 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/4bc46f16287842ac8ed387e9fce2fec1, entries=150, sequenceid=353, filesize=11.9 K 2023-04-28 14:58:12,088 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~344.72 KB/352992, currentSize=60.38 KB/61830 for f40347916069de574ae857f0751743f8 in 912ms, sequenceid=353, compaction requested=false 2023-04-28 14:58:12,088 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:12,089 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:12,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:12,089 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:12,089 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:12,089 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:12,089 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:12,089 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:12,090 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:12,090 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:12,090 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:12,090 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:12,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:12,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:12,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:12,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:12,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:12,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:12,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:12,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:12,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:12,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:12,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:12,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:12,092 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:12,092 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:12,093 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:12,093 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:12,093 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:12,094 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:12,094 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:12,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:12,094 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:12,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:12,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:12,095 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:12,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:12,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:12,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:12,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:12,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:12,096 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:12,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:12,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:12,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:12,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:12,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:12,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:12,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:12,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:12,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:12,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:12,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:12,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:12,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:12,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:12,099 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:12,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:12,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:12,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:12,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:12,099 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:12,099 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:12,199 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:12,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:12,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:12,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:12,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:12,202 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:12,202 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:12,202 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:12,202 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:12,202 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:12,202 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:12,203 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:12,203 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:12,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:12,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:12,203 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:12,203 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:12,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:12,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:12,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:12,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:12,204 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:12,204 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:12,204 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:12,204 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:12,204 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:12,204 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:12,205 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:12,205 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:12,205 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:12,205 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:12,205 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f40347916069de574ae857f0751743f8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-28 14:58:12,205 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=A 2023-04-28 14:58:12,205 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:12,205 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=B 2023-04-28 14:58:12,205 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:12,206 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=C 2023-04-28 14:58:12,206 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:12,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on f40347916069de574ae857f0751743f8 2023-04-28 14:58:12,328 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:12,380 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:12,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50270 deadline: 1682693952379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:12,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:12,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50270 deadline: 1682693952481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:12,628 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/7f125540dcdd4e06ada696b564098d06 2023-04-28 14:58:12,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:12,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50270 deadline: 1682693952684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:12,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:12,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50270 deadline: 1682693952989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:13,048 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/b0eb649b8d1e4fbca9932f731840a4c3 2023-04-28 14:58:13,457 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/e54fa757d8064e66830ce7350f7b371e 2023-04-28 14:58:13,460 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/7f125540dcdd4e06ada696b564098d06 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/7f125540dcdd4e06ada696b564098d06 2023-04-28 14:58:13,463 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/7f125540dcdd4e06ada696b564098d06, entries=100, sequenceid=368, filesize=9.5 K 2023-04-28 14:58:13,464 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/b0eb649b8d1e4fbca9932f731840a4c3 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/b0eb649b8d1e4fbca9932f731840a4c3 2023-04-28 14:58:13,468 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/b0eb649b8d1e4fbca9932f731840a4c3, entries=100, sequenceid=368, filesize=9.5 K 2023-04-28 14:58:13,468 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/e54fa757d8064e66830ce7350f7b371e as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/e54fa757d8064e66830ce7350f7b371e 2023-04-28 14:58:13,471 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/e54fa757d8064e66830ce7350f7b371e, entries=100, sequenceid=368, filesize=9.5 K 2023-04-28 14:58:13,472 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for f40347916069de574ae857f0751743f8 in 1267ms, sequenceid=368, compaction requested=true 2023-04-28 14:58:13,472 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:13,472 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:13,472 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:13,472 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:13,472 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:13,472 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:13,472 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:13,473 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:13,473 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:13,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:13,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:13,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:13,473 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:13,474 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:13,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:13,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:13,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:13,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:13,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:13,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:13,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d550886[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:13,475 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:13,475 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:13,475 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:13,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:13,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:13,475 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:13,476 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:13,476 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:13,476 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:13,476 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:13,476 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:13,476 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:13,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:13,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:13,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:13,477 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:13,477 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:13,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:13,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:13,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:13,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:13,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:13,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:13,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:13,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:13,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:13,479 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:13,479 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:13,479 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:13,479 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:13,479 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:13,480 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:13,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:13,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:13,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:13,480 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:13,480 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:13,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:13,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:13,481 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:13,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:13,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:13,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:13,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:13,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:13,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:13,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:13,482 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:13,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:13,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:13,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:13,483 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:13,483 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:13,483 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:13,483 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:13,483 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:13,483 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:13,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:13,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:13,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:13,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:13,483 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:13,484 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:13,484 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:13,484 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:13,484 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:13,484 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:13,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:13,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:13,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:13,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on f40347916069de574ae857f0751743f8 2023-04-28 14:58:13,493 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f40347916069de574ae857f0751743f8 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-28 14:58:13,493 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=A 2023-04-28 14:58:13,493 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:13,493 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=B 2023-04-28 14:58:13,493 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:13,493 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=C 2023-04-28 14:58:13,493 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:13,512 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=389 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/a6fcc77810a64359ade5b8283483cdb1 2023-04-28 14:58:13,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:13,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50270 deadline: 1682693953519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:13,525 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=389 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/68816356eb4144fdabd4dcb9e82a0b56 2023-04-28 14:58:13,537 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=389 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/8c2e25a06a1e402d8560d77d242e63ad 2023-04-28 14:58:13,541 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/a6fcc77810a64359ade5b8283483cdb1 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/a6fcc77810a64359ade5b8283483cdb1 2023-04-28 14:58:13,545 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/a6fcc77810a64359ade5b8283483cdb1, entries=200, sequenceid=389, filesize=14.3 K 2023-04-28 14:58:13,546 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/68816356eb4144fdabd4dcb9e82a0b56 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/68816356eb4144fdabd4dcb9e82a0b56 2023-04-28 14:58:13,555 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/68816356eb4144fdabd4dcb9e82a0b56, entries=150, sequenceid=389, filesize=11.9 K 2023-04-28 14:58:13,556 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/8c2e25a06a1e402d8560d77d242e63ad as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/8c2e25a06a1e402d8560d77d242e63ad 2023-04-28 14:58:13,560 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/8c2e25a06a1e402d8560d77d242e63ad, entries=150, sequenceid=389, filesize=11.9 K 2023-04-28 14:58:13,561 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for f40347916069de574ae857f0751743f8 in 68ms, sequenceid=389, compaction requested=true 2023-04-28 14:58:13,561 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:13,561 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:13,561 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:13,561 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:58:13,561 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:58:13,561 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:13,564 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51474 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:58:13,565 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): f40347916069de574ae857f0751743f8/A is initiating minor compaction (all files) 2023-04-28 14:58:13,565 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46594 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:58:13,565 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f40347916069de574ae857f0751743f8/A in TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:13,565 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): f40347916069de574ae857f0751743f8/B is initiating minor compaction (all files) 2023-04-28 14:58:13,565 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f40347916069de574ae857f0751743f8/B in TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:13,565 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/5f3b4fd07a2a4f89bcc94c29c5858265, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/e5c0df22792a413b8dda6341feaafd5a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/7f125540dcdd4e06ada696b564098d06, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/a6fcc77810a64359ade5b8283483cdb1] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp, totalSize=50.3 K 2023-04-28 14:58:13,565 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/4ad2bb2074a2413e9325584276381b30, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/0d38f92b82664883a5b0eaaaf66ccb0c, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/b0eb649b8d1e4fbca9932f731840a4c3, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/68816356eb4144fdabd4dcb9e82a0b56] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp, totalSize=45.5 K 2023-04-28 14:58:13,565 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 5f3b4fd07a2a4f89bcc94c29c5858265, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1682693890613 2023-04-28 14:58:13,565 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 4ad2bb2074a2413e9325584276381b30, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1682693890613 2023-04-28 14:58:13,566 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting e5c0df22792a413b8dda6341feaafd5a, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1682693891151 2023-04-28 14:58:13,566 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 0d38f92b82664883a5b0eaaaf66ccb0c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1682693891151 2023-04-28 14:58:13,566 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 7f125540dcdd4e06ada696b564098d06, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1682693891199 2023-04-28 14:58:13,566 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting b0eb649b8d1e4fbca9932f731840a4c3, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1682693891199 2023-04-28 14:58:13,567 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting a6fcc77810a64359ade5b8283483cdb1, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1682693892360 2023-04-28 14:58:13,567 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 68816356eb4144fdabd4dcb9e82a0b56, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1682693892360 2023-04-28 14:58:13,578 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): f40347916069de574ae857f0751743f8#A#compaction#231 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:13,584 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:13,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:13,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:13,602 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): f40347916069de574ae857f0751743f8#B#compaction#232 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:13,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:13,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:13,602 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:13,602 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:13,602 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:13,602 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:13,602 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:13,602 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:13,604 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:13,604 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:13,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:13,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:13,604 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:13,604 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:13,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:13,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:13,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:13,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:13,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:13,606 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:13,606 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:13,606 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:13,606 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:13,606 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:13,606 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:13,606 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:13,606 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:13,606 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:13,606 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f40347916069de574ae857f0751743f8 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-28 14:58:13,606 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=A 2023-04-28 14:58:13,607 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:13,607 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=B 2023-04-28 14:58:13,607 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:13,607 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=C 2023-04-28 14:58:13,607 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:13,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on f40347916069de574ae857f0751743f8 2023-04-28 14:58:13,626 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:13,702 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:13,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:13,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693953677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:13,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693953680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:13,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9554, totalHeapSize=21976, min timestamp=1682693893674, max timestamp=1682693893684Num uniques 859; 2023-04-28 14:58:13,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=39324, new segment data size=9554, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:58:13,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9554, totalHeapSize=21976, min timestamp=1682693893674, max timestamp=1682693893684Num uniques 859; 2023-04-28 14:58:13,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=160, cellSize=7320, totalHeapSize=16880, min timestamp=1682693893674, max timestamp=1682693893684Num uniques 859; 2023-04-28 14:58:13,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=39324, new segment data size=9554, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:58:13,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=7320, suffix heap size=117064,new segment heap size=16880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=160 2023-04-28 14:58:13,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:13,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693953724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:13,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:13,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693953727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:13,730 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:13,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50270 deadline: 1682693953728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:13,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:13,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693953803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:13,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:13,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693953804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:13,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:13,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693953828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:13,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:13,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693953831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:13,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:13,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50270 deadline: 1682693953831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:13,995 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/664c55fb574042ca9bf017a47a90e530 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/664c55fb574042ca9bf017a47a90e530 2023-04-28 14:58:13,998 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f40347916069de574ae857f0751743f8/A of f40347916069de574ae857f0751743f8 into 664c55fb574042ca9bf017a47a90e530(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:13,998 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:13,999 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8., storeName=f40347916069de574ae857f0751743f8/A, priority=12, startTime=1682693893561; duration=0sec 2023-04-28 14:58:13,999 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:13,999 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:58:14,000 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46594 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:58:14,000 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): f40347916069de574ae857f0751743f8/C is initiating minor compaction (all files) 2023-04-28 14:58:14,000 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f40347916069de574ae857f0751743f8/C in TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:14,000 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/5b6c5c9671554807b0b64c75604e9e78, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/4bc46f16287842ac8ed387e9fce2fec1, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/e54fa757d8064e66830ce7350f7b371e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/8c2e25a06a1e402d8560d77d242e63ad] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp, totalSize=45.5 K 2023-04-28 14:58:14,000 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 5b6c5c9671554807b0b64c75604e9e78, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1682693890613 2023-04-28 14:58:14,001 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 4bc46f16287842ac8ed387e9fce2fec1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1682693891151 2023-04-28 14:58:14,001 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting e54fa757d8064e66830ce7350f7b371e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1682693891199 2023-04-28 14:58:14,001 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 8c2e25a06a1e402d8560d77d242e63ad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1682693892360 2023-04-28 14:58:14,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:14,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693954007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,008 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): f40347916069de574ae857f0751743f8#C#compaction#234 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:14,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:14,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693954008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,017 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/4b2245ebb6e2473796815bc1926ad140 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/4b2245ebb6e2473796815bc1926ad140 2023-04-28 14:58:14,022 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f40347916069de574ae857f0751743f8/B of f40347916069de574ae857f0751743f8 into 4b2245ebb6e2473796815bc1926ad140(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:14,022 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:14,022 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8., storeName=f40347916069de574ae857f0751743f8/B, priority=12, startTime=1682693893561; duration=0sec 2023-04-28 14:58:14,022 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:14,023 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/655edcf8431b455a9b840681bff3388d as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/655edcf8431b455a9b840681bff3388d 2023-04-28 14:58:14,024 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/a13ba8ca6dd54a569915e38d977706c3 2023-04-28 14:58:14,028 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f40347916069de574ae857f0751743f8/C of f40347916069de574ae857f0751743f8 into 655edcf8431b455a9b840681bff3388d(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:14,028 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:14,028 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8., storeName=f40347916069de574ae857f0751743f8/C, priority=12, startTime=1682693893561; duration=0sec 2023-04-28 14:58:14,028 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:14,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:14,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693954031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,036 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:14,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50270 deadline: 1682693954034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,036 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:14,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693954035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:14,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693954308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:14,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693954311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,335 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:14,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693954335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,337 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:14,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50270 deadline: 1682693954337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,340 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:14,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693954339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,449 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/c8dae1cf5d444790a593d628800579ae 2023-04-28 14:58:14,459 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/c6cd7fea1aa24292936a77093ec3354e 2023-04-28 14:58:14,462 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/a13ba8ca6dd54a569915e38d977706c3 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/a13ba8ca6dd54a569915e38d977706c3 2023-04-28 14:58:14,466 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/a13ba8ca6dd54a569915e38d977706c3, entries=150, sequenceid=400, filesize=11.9 K 2023-04-28 14:58:14,466 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/c8dae1cf5d444790a593d628800579ae as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/c8dae1cf5d444790a593d628800579ae 2023-04-28 14:58:14,470 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/c8dae1cf5d444790a593d628800579ae, entries=150, sequenceid=400, filesize=11.9 K 2023-04-28 14:58:14,470 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/c6cd7fea1aa24292936a77093ec3354e as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/c6cd7fea1aa24292936a77093ec3354e 2023-04-28 14:58:14,473 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/c6cd7fea1aa24292936a77093ec3354e, entries=150, sequenceid=400, filesize=11.9 K 2023-04-28 14:58:14,474 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=131.99 KB/135156 for f40347916069de574ae857f0751743f8 in 868ms, sequenceid=400, compaction requested=false 2023-04-28 14:58:14,474 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:14,474 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:14,474 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:14,474 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:14,474 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,475 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:14,475 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:14,476 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:14,476 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:14,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:14,476 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:14,476 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:14,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:14,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:14,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:14,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:14,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:14,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61a1df11[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:14,478 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:14,478 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:14,478 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:14,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:14,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:14,478 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:14,478 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:14,478 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:14,478 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,478 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:14,479 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,479 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:14,479 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:14,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:14,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:14,479 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:14,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:14,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:14,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:14,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:14,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:14,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:14,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:14,481 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:14,481 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:14,481 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:14,481 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:14,481 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:14,482 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:14,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:14,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:14,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:14,482 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:14,482 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:14,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:14,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:14,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:14,482 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:14,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:14,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:14,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:14,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:14,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:14,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,484 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:14,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:14,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,486 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,486 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:14,486 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:14,486 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,486 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:14,486 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:14,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:14,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:14,486 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:14,486 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:14,487 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:14,487 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:14,487 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:14,487 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:14,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:14,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:14,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:14,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:14,587 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:14,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:14,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:14,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:14,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:14,590 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:14,590 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:14,590 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:14,590 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:14,590 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:14,590 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:14,591 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:14,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:14,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:14,591 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:14,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:14,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:14,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:14,592 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:14,593 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,593 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:14,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:14,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:14,604 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:14,604 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:14,604 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:14,605 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:14,605 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:14,605 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:14,605 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:14,605 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f40347916069de574ae857f0751743f8 3/3 column families, dataSize=131.99 KB heapSize=375.32 KB 2023-04-28 14:58:14,605 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=A 2023-04-28 14:58:14,605 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:14,605 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=B 2023-04-28 14:58:14,606 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:14,606 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=C 2023-04-28 14:58:14,606 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:14,614 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=439 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/d5b12b1efa8541b79e6ccfe88fa2bcb2 2023-04-28 14:58:14,622 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=439 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/788b095fb93a4a2e91e2f228586503e1 2023-04-28 14:58:14,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on f40347916069de574ae857f0751743f8 2023-04-28 14:58:14,815 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:14,830 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:14,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693954827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,831 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:14,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693954828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,839 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:14,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693954838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:14,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50270 deadline: 1682693954839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,843 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:14,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693954841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:14,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693954931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:14,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:14,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693954932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,031 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.54 KB at sequenceid=439 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/027700f694d74253ba9f3a3994e1a491 2023-04-28 14:58:15,034 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/d5b12b1efa8541b79e6ccfe88fa2bcb2 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/d5b12b1efa8541b79e6ccfe88fa2bcb2 2023-04-28 14:58:15,037 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/d5b12b1efa8541b79e6ccfe88fa2bcb2, entries=150, sequenceid=439, filesize=11.9 K 2023-04-28 14:58:15,038 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/788b095fb93a4a2e91e2f228586503e1 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/788b095fb93a4a2e91e2f228586503e1 2023-04-28 14:58:15,041 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/788b095fb93a4a2e91e2f228586503e1, entries=150, sequenceid=439, filesize=11.9 K 2023-04-28 14:58:15,042 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/027700f694d74253ba9f3a3994e1a491 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/027700f694d74253ba9f3a3994e1a491 2023-04-28 14:58:15,044 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/027700f694d74253ba9f3a3994e1a491, entries=150, sequenceid=439, filesize=11.9 K 2023-04-28 14:58:15,045 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.99 KB/135156, heapSize ~375.27 KB/384280, currentSize=53.67 KB/54960 for f40347916069de574ae857f0751743f8 in 440ms, sequenceid=439, compaction requested=true 2023-04-28 14:58:15,045 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:15,045 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:15,045 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:15,045 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:15,045 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,045 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:15,045 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:15,046 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,046 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:15,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:15,046 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,046 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:15,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:15,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:15,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:15,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:15,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41f81068[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:15,048 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:15,048 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,049 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,049 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:15,049 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:15,049 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,049 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:15,049 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:15,050 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,050 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:15,050 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:15,050 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:15,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:15,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:15,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:15,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:15,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:15,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:15,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:15,052 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:15,052 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:15,052 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:15,052 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:15,052 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:15,053 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,053 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:15,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:15,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,054 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:15,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:15,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:15,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:15,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,054 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:15,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,055 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:15,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,057 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,057 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,057 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:15,057 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:15,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:15,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:15,057 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:15,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:15,057 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:15,057 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:15,057 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,057 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:15,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:15,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:15,057 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,057 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,057 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on f40347916069de574ae857f0751743f8 2023-04-28 14:58:15,137 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f40347916069de574ae857f0751743f8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-28 14:58:15,138 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=A 2023-04-28 14:58:15,138 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:15,138 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=B 2023-04-28 14:58:15,138 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:15,138 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=C 2023-04-28 14:58:15,138 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:15,148 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/0b479bc616e14ddda34dbd7c544ef430 2023-04-28 14:58:15,157 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:15,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:15,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:15,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:15,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:15,160 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:15,160 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:15,160 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:15,160 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:15,161 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,161 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,161 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:15,161 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:15,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:15,162 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,162 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:15,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:15,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,162 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:15,162 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:15,162 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:15,162 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:15,162 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:15,162 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:15,163 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:15,163 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:15,163 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:15,163 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:15,163 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:15,163 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:15,163 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:15,163 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:15,163 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,163 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:15,163 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:15,164 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,164 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:15,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:15,164 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:15,164 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:15,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:15,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:15,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:15,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43c8f50b[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:15,165 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:15,165 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,166 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,166 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:15,166 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,166 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:15,166 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:15,166 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:15,167 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,167 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:15,167 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:15,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:15,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:15,167 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:15,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:15,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:15,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:15,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:15,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:15,169 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:15,169 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:15,169 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:15,169 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:15,169 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:15,170 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,170 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,170 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:15,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:15,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:15,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:15,170 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:15,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:15,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:15,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,171 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:15,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,172 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:15,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693955170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,172 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:15,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693955170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,173 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,173 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,173 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,173 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,173 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:15,173 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:15,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:15,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:15,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:15,173 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:15,173 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:15,173 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,173 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:15,173 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,173 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:15,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:15,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:15,273 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:15,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:15,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:15,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:15,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693955273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:15,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693955273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:15,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:15,276 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:15,276 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:15,276 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:15,276 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:15,276 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,276 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,277 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:15,277 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:15,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:15,277 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,277 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:15,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:15,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,278 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:15,279 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:15,279 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:15,279 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:15,279 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:15,279 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:15,279 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:15,279 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:15,279 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:15,279 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:15,279 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:15,279 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:15,279 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:15,279 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:15,279 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,279 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:15,279 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:15,280 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,280 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:15,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:15,280 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,280 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:15,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:15,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:15,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:15,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:15,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@160cd4cc[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:15,281 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:15,281 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,282 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,282 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:15,282 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:15,282 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:15,282 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,282 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:15,283 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,283 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:15,283 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:15,283 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:15,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:15,285 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:15,285 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:15,285 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:15,285 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:15,285 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:15,286 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:15,286 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,286 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:15,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:15,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:15,286 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:15,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,287 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:15,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,289 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,289 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,289 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,289 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:15,289 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:15,289 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:15,289 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:15,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:15,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:15,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:15,289 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,289 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,290 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,290 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:15,290 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:15,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:15,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:15,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:15,390 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:15,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:15,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:15,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:15,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:15,392 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:15,392 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:15,392 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:15,392 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:15,393 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,393 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,393 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:15,393 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:15,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:15,393 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,394 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:15,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:15,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,394 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:15,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:15,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:15,396 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:15,396 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:15,396 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:15,397 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:15,397 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:15,397 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:15,397 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:15,397 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:15,397 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:15,397 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:15,397 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:15,397 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,397 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:15,397 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:15,398 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,398 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:15,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:15,398 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,398 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:15,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:15,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:15,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:15,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:15,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@793ac84b[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:15,399 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:15,399 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,400 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,400 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:15,400 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:15,400 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:15,400 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,400 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:15,401 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,401 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:15,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:15,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:15,401 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:15,401 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:15,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:15,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:15,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:15,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:15,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:15,403 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:15,403 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:15,403 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:15,403 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:15,403 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:15,404 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:15,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:15,404 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,404 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:15,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:15,404 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:15,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:15,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:15,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:15,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,405 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,407 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,407 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:15,407 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:15,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:15,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:15,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:15,407 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:15,407 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:15,407 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,407 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,407 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:15,407 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,407 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:15,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:15,407 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:15,407 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,477 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:15,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693955477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:15,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693955477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,507 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:15,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:15,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:15,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:15,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:15,510 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:15,510 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:15,510 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:15,510 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:15,511 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,511 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,511 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:15,511 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:15,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:15,512 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,512 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:15,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:15,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,512 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:15,512 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:15,512 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:15,512 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:15,512 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:15,512 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:15,513 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:15,513 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:15,513 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:15,513 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:15,513 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:15,513 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:15,513 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:15,513 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:15,513 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,513 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:15,513 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:15,514 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,514 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:15,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:15,514 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,514 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:15,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:15,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:15,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:15,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:15,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7840fc54[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:15,515 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:15,515 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,516 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,516 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:15,516 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:15,516 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,516 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:15,516 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:15,517 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:15,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:15,517 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:15,517 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:15,517 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:15,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:15,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:15,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:15,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:15,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:15,519 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:15,519 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:15,519 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:15,519 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:15,519 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:15,519 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,520 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:15,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:15,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,520 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:15,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:15,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:15,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:15,520 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:15,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,522 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:15,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,523 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,524 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,524 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,524 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,524 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:15,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:15,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:15,524 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:15,524 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,524 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,524 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:15,524 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:15,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:15,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:15,524 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:15,524 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:15,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:15,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:15,561 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/52ed6e1211424dd2a8d9fa4b1095e3c1 2023-04-28 14:58:15,624 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:15,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:15,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:15,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:15,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:15,627 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:15,627 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:15,627 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:15,627 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:15,627 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,627 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,628 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:15,628 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:15,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:15,628 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,628 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:15,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:15,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:15,629 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:15,629 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:15,629 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:15,629 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:15,629 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:15,629 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:15,629 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:15,629 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:15,629 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:15,629 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:15,629 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:15,629 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:15,629 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,630 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:15,630 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:15,630 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,630 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:15,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:15,630 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,630 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:15,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:15,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:15,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:15,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:15,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c242efa[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:15,632 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:15,632 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,632 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,632 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:15,632 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,632 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:15,632 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:15,632 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:15,633 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:15,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:15,633 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:15,633 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:15,633 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:15,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:15,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:15,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:15,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:15,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:15,635 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:15,635 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:15,635 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:15,635 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:15,635 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:15,636 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,636 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:15,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:15,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,636 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:15,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:15,636 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:15,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:15,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:15,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,637 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:15,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,639 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,639 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:15,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:15,639 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,639 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:15,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:15,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:15,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:15,639 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:15,639 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,639 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,639 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:15,639 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,639 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:15,639 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:15,639 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:15,739 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:15,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:15,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:15,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:15,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:15,742 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:15,742 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:15,742 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:15,742 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:15,742 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,742 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,744 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:15,744 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:15,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:15,744 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,744 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:15,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:15,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,745 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:15,745 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:15,745 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:15,745 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:15,745 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:15,745 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:15,745 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:15,745 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:15,745 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:15,745 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:15,745 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:15,745 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:15,746 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:15,746 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:15,746 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,746 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:15,746 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:15,746 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,746 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:15,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:15,747 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,747 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:15,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:15,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:15,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:15,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:15,748 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:15,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c7e2f06[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:15,748 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,748 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,748 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:15,748 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:15,748 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:15,748 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,749 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:15,749 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,749 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:15,749 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:15,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:15,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:15,749 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:15,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:15,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:15,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:15,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:15,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:15,751 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:15,751 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:15,752 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:15,752 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:15,752 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:15,752 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:15,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:15,752 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,753 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:15,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:15,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:15,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:15,753 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:15,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,754 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:15,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,756 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,756 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,756 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,756 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,756 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,756 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:15,756 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:15,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:15,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:15,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:15,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:15,756 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:15,756 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:15,756 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,756 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:15,756 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:15,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:15,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:15,782 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:15,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50218 deadline: 1682693955781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,782 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:15,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50300 deadline: 1682693955781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:15,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693955843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:15,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50270 deadline: 1682693955844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,850 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:15,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693955848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,857 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:15,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:15,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:15,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:15,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:15,860 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:15,860 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:15,860 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:15,860 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:15,860 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,860 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,861 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:15,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:15,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:15,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,862 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:15,862 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,862 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:15,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:15,863 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:15,863 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:15,863 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:15,863 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:15,863 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:15,863 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:15,864 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:15,864 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:15,864 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:15,864 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:15,864 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:15,864 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:15,864 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:15,864 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,864 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:15,864 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:15,865 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,865 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:15,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:15,865 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,865 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:15,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:15,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:15,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:15,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:15,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2329c652[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:15,868 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:15,868 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,869 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,869 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:15,869 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:15,869 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:15,869 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,869 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:15,870 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:15,870 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:15,870 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,870 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:15,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:15,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:15,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:15,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:15,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:15,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:15,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:15,874 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:15,874 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:15,874 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:15,874 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:15,874 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:15,876 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,876 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:15,876 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:15,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:15,876 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:15,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:15,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,877 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:15,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:15,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,879 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,879 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,879 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,879 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:15,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:15,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:15,879 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:15,879 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:15,880 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,880 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:15,880 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:15,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:15,880 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:15,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:15,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:15,880 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:15,880 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:15,971 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/292aa52d0a5f4e89b8243071aee3d2ca 2023-04-28 14:58:15,974 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/0b479bc616e14ddda34dbd7c544ef430 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/0b479bc616e14ddda34dbd7c544ef430 2023-04-28 14:58:15,977 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/0b479bc616e14ddda34dbd7c544ef430, entries=200, sequenceid=452, filesize=14.3 K 2023-04-28 14:58:15,978 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/52ed6e1211424dd2a8d9fa4b1095e3c1 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/52ed6e1211424dd2a8d9fa4b1095e3c1 2023-04-28 14:58:15,980 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:15,980 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/52ed6e1211424dd2a8d9fa4b1095e3c1, entries=150, sequenceid=452, filesize=11.9 K 2023-04-28 14:58:15,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:15,981 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/292aa52d0a5f4e89b8243071aee3d2ca as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/292aa52d0a5f4e89b8243071aee3d2ca 2023-04-28 14:58:15,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:15,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:15,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:15,986 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:15,986 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:15,986 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:15,986 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:15,986 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,986 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,989 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,989 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:15,989 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/292aa52d0a5f4e89b8243071aee3d2ca, entries=150, sequenceid=452, filesize=11.9 K 2023-04-28 14:58:15,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:15,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:15,989 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:15,989 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:15,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,989 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for f40347916069de574ae857f0751743f8 in 852ms, sequenceid=452, compaction requested=true 2023-04-28 14:58:15,990 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:15,990 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:15,990 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:58:15,990 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:58:15,990 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:15,990 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:15,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:15,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:15,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:15,990 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:15,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:15,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:15,991 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:15,991 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:15,991 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:15,991 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:15,991 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:58:15,991 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:15,991 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:15,991 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): f40347916069de574ae857f0751743f8/B is initiating minor compaction (all files) 2023-04-28 14:58:15,991 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f40347916069de574ae857f0751743f8/B in TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:15,991 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/4b2245ebb6e2473796815bc1926ad140, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/c8dae1cf5d444790a593d628800579ae, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/788b095fb93a4a2e91e2f228586503e1, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/52ed6e1211424dd2a8d9fa4b1095e3c1] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp, totalSize=48.2 K 2023-04-28 14:58:15,991 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:15,991 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51764 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:58:15,992 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): f40347916069de574ae857f0751743f8/A is initiating minor compaction (all files) 2023-04-28 14:58:15,992 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f40347916069de574ae857f0751743f8/A in TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:15,992 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/664c55fb574042ca9bf017a47a90e530, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/a13ba8ca6dd54a569915e38d977706c3, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/d5b12b1efa8541b79e6ccfe88fa2bcb2, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/0b479bc616e14ddda34dbd7c544ef430] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp, totalSize=50.6 K 2023-04-28 14:58:15,992 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 4b2245ebb6e2473796815bc1926ad140, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1682693892360 2023-04-28 14:58:15,991 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f40347916069de574ae857f0751743f8 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-28 14:58:15,992 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 664c55fb574042ca9bf017a47a90e530, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1682693892360 2023-04-28 14:58:15,992 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=A 2023-04-28 14:58:15,992 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting c8dae1cf5d444790a593d628800579ae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1682693893506 2023-04-28 14:58:15,992 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting a13ba8ca6dd54a569915e38d977706c3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1682693893506 2023-04-28 14:58:15,992 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:15,992 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=B 2023-04-28 14:58:15,992 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:15,993 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=C 2023-04-28 14:58:15,993 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting d5b12b1efa8541b79e6ccfe88fa2bcb2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1682693893719 2023-04-28 14:58:15,993 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 788b095fb93a4a2e91e2f228586503e1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1682693893719 2023-04-28 14:58:15,993 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:15,993 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 0b479bc616e14ddda34dbd7c544ef430, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1682693894822 2023-04-28 14:58:15,993 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 52ed6e1211424dd2a8d9fa4b1095e3c1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1682693894823 2023-04-28 14:58:15,999 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/e3a0bc88f2734554bb553a1594e44c7c 2023-04-28 14:58:16,001 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): f40347916069de574ae857f0751743f8#B#compaction#244 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:16,001 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): f40347916069de574ae857f0751743f8#A#compaction#245 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:16,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on f40347916069de574ae857f0751743f8 2023-04-28 14:58:16,285 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:16,307 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:16,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50300 deadline: 1682693956305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,308 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:16,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50218 deadline: 1682693956306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,408 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:16,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50300 deadline: 1682693956408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,409 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/7db1ba1de2dd473183e226ad49a8966d as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/7db1ba1de2dd473183e226ad49a8966d 2023-04-28 14:58:16,410 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/87d7ca87a0a941e8924b9df1eb907707 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/87d7ca87a0a941e8924b9df1eb907707 2023-04-28 14:58:16,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:16,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50218 deadline: 1682693956409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,412 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/337426151a5c49f689052d8dbc9df36b 2023-04-28 14:58:16,414 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f40347916069de574ae857f0751743f8/B of f40347916069de574ae857f0751743f8 into 7db1ba1de2dd473183e226ad49a8966d(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:16,414 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:16,414 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8., storeName=f40347916069de574ae857f0751743f8/B, priority=12, startTime=1682693895990; duration=0sec 2023-04-28 14:58:16,414 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:16,414 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:58:16,415 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f40347916069de574ae857f0751743f8/A of f40347916069de574ae857f0751743f8 into 87d7ca87a0a941e8924b9df1eb907707(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:16,415 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:58:16,415 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:16,416 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): f40347916069de574ae857f0751743f8/C is initiating minor compaction (all files) 2023-04-28 14:58:16,416 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8., storeName=f40347916069de574ae857f0751743f8/A, priority=12, startTime=1682693895990; duration=0sec 2023-04-28 14:58:16,416 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f40347916069de574ae857f0751743f8/C in TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:16,416 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:16,416 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/655edcf8431b455a9b840681bff3388d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/c6cd7fea1aa24292936a77093ec3354e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/027700f694d74253ba9f3a3994e1a491, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/292aa52d0a5f4e89b8243071aee3d2ca] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp, totalSize=48.2 K 2023-04-28 14:58:16,416 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 655edcf8431b455a9b840681bff3388d, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1682693892360 2023-04-28 14:58:16,416 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting c6cd7fea1aa24292936a77093ec3354e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1682693893506 2023-04-28 14:58:16,417 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 027700f694d74253ba9f3a3994e1a491, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1682693893719 2023-04-28 14:58:16,417 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 292aa52d0a5f4e89b8243071aee3d2ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1682693894823 2023-04-28 14:58:16,426 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): f40347916069de574ae857f0751743f8#C#compaction#247 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:16,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:16,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50300 deadline: 1682693956609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,617 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:16,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50218 deadline: 1682693956614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,835 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/d3665f247c9a4e53a10b851c670946e2 2023-04-28 14:58:16,839 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/6ee09e2ef88a4cf5963ebf6c2fca9edc as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/6ee09e2ef88a4cf5963ebf6c2fca9edc 2023-04-28 14:58:16,839 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/e3a0bc88f2734554bb553a1594e44c7c as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/e3a0bc88f2734554bb553a1594e44c7c 2023-04-28 14:58:16,842 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/e3a0bc88f2734554bb553a1594e44c7c, entries=150, sequenceid=471, filesize=11.9 K 2023-04-28 14:58:16,843 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/337426151a5c49f689052d8dbc9df36b as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/337426151a5c49f689052d8dbc9df36b 2023-04-28 14:58:16,843 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f40347916069de574ae857f0751743f8/C of f40347916069de574ae857f0751743f8 into 6ee09e2ef88a4cf5963ebf6c2fca9edc(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:16,843 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:16,843 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8., storeName=f40347916069de574ae857f0751743f8/C, priority=12, startTime=1682693895990; duration=0sec 2023-04-28 14:58:16,844 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:16,847 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/337426151a5c49f689052d8dbc9df36b, entries=150, sequenceid=471, filesize=11.9 K 2023-04-28 14:58:16,848 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/d3665f247c9a4e53a10b851c670946e2 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/d3665f247c9a4e53a10b851c670946e2 2023-04-28 14:58:16,851 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/d3665f247c9a4e53a10b851c670946e2, entries=150, sequenceid=471, filesize=11.9 K 2023-04-28 14:58:16,851 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for f40347916069de574ae857f0751743f8 in 860ms, sequenceid=471, compaction requested=false 2023-04-28 14:58:16,851 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:16,852 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:16,852 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:16,852 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:16,852 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,852 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:16,852 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:16,852 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,853 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:16,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:16,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:16,853 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:16,853 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:16,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:16,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:16,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:16,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:16,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:16,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c743c3f[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:16,854 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:16,854 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:16,855 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:16,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:16,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:16,855 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:16,855 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:16,855 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:16,855 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,855 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:16,856 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,856 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:16,856 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:16,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:16,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:16,856 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:16,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:16,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:16,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:16,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:16,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:16,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:16,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:16,858 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:16,858 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:16,858 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:16,858 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:16,858 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:16,859 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:16,859 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:16,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:16,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:16,859 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:16,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:16,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:16,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:16,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:16,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:16,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:16,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:16,860 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:16,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:16,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:16,861 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:16,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:16,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,862 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,862 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:16,862 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:16,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:16,862 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:16,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:16,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:16,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:16,862 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:16,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:16,863 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:16,863 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:16,863 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:16,863 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,863 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:16,863 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:16,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:16,863 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:16,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:16,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on f40347916069de574ae857f0751743f8 2023-04-28 14:58:16,914 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f40347916069de574ae857f0751743f8 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-28 14:58:16,914 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=A 2023-04-28 14:58:16,914 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:16,914 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=B 2023-04-28 14:58:16,914 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:16,914 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=C 2023-04-28 14:58:16,914 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:16,921 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/b3b7d05686a74d62b2e7e8bd96c4a927 2023-04-28 14:58:16,936 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:16,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50218 deadline: 1682693956934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,937 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:16,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50300 deadline: 1682693956936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,963 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:16,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:16,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:16,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:16,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:16,965 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:16,965 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:16,965 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:16,965 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:16,966 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:16,966 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:16,966 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,966 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:16,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:16,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:16,966 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,967 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:16,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:16,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:16,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:16,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:16,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:16,968 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:16,968 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:16,968 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:16,968 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:16,968 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:16,968 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:16,968 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:16,968 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:16,968 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:16,968 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:16,968 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:16,968 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:16,968 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:16,968 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,968 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:16,968 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:16,970 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,970 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:16,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:16,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:16,970 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:16,970 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:16,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:16,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:16,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:16,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:16,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:16,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fa8ab5[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:16,971 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:16,971 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:16,972 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:16,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:16,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:16,972 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,972 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:16,972 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:16,972 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:16,972 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:16,973 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,973 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:16,973 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:16,973 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:16,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:16,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:16,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:16,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:16,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:16,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:16,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:16,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:16,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:16,975 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:16,976 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:16,976 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:16,976 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:16,976 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:16,979 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:16,979 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:16,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:16,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:16,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:16,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:16,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:16,981 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:16,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:16,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:16,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:16,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:16,982 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:16,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:16,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:16,985 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:16,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:16,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,987 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:16,987 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,987 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:16,987 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:16,987 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:16,987 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:16,987 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:16,987 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:16,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:16,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:16,987 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:16,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:16,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:16,988 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:16,988 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:16,987 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:16,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:16,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:16,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:17,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:17,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50218 deadline: 1682693957037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,040 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:17,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50300 deadline: 1682693957038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,088 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:17,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:17,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:17,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:17,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:17,090 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:17,090 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:17,090 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:17,090 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:17,091 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:17,091 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:17,091 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,091 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:17,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:17,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:17,092 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,092 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:17,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:17,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:17,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:17,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:17,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:17,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:17,093 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:17,093 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:17,093 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:17,093 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:17,093 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:17,093 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:17,093 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:17,093 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:17,093 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:17,093 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:17,093 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:17,093 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:17,093 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,093 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:17,093 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:17,094 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,094 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:17,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:17,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:17,095 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:17,095 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:17,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:17,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:17,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:17,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:17,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:17,096 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:17,096 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:17,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c019393[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:17,097 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:17,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:17,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:17,097 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:17,097 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:17,097 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:17,097 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,097 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:17,097 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,097 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:17,097 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:17,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:17,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:17,097 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:17,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:17,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:17,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:17,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:17,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:17,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:17,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:17,099 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:17,099 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:17,099 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:17,099 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:17,099 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:17,100 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:17,100 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:17,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:17,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:17,100 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:17,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:17,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:17,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:17,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:17,101 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:17,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:17,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:17,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:17,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:17,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:17,102 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:17,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:17,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,103 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,103 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:17,103 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:17,103 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:17,103 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:17,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:17,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:17,103 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,103 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:17,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:17,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:17,104 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:17,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:17,104 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:17,104 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:17,104 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:17,104 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:17,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:17,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:17,204 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:17,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:17,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:17,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:17,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:17,207 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:17,208 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:17,208 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:17,208 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:17,208 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:17,208 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:17,209 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,209 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:17,209 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,209 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:17,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:17,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:17,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:17,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:17,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:17,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:17,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:17,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:17,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:17,210 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:17,210 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:17,210 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:17,214 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:17,214 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:17,214 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:17,214 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:17,214 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:17,214 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:17,214 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:17,214 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:17,214 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,214 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:17,214 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:17,215 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,215 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:17,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:17,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:17,216 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:17,216 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:17,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:17,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:17,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:17,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:17,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:17,217 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:17,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1177c117[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:17,217 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:17,217 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:17,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:17,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:17,217 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:17,218 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:17,218 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:17,218 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,218 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:17,218 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,218 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:17,218 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:17,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:17,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:17,218 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:17,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:17,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:17,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:17,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:17,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:17,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:17,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:17,221 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:17,221 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:17,221 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:17,221 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:17,221 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:17,222 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:17,222 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:17,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:17,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:17,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:17,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:17,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:17,222 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:17,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:17,222 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:17,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:17,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:17,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:17,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:17,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:17,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,224 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:17,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:17,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,225 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:17,225 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:17,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:17,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:17,225 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:17,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:17,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:17,225 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:17,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:17,226 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:17,225 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:17,226 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:17,226 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:17,226 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,226 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:17,226 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:17,226 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:17,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:17,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50218 deadline: 1682693957240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,243 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:17,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50300 deadline: 1682693957242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,326 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:17,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:17,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:17,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:17,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:17,328 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:17,328 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:17,328 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:17,328 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:17,329 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:17,329 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:17,329 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,329 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:17,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:17,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:17,330 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,330 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:17,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:17,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:17,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:17,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:17,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:17,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:17,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:17,330 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:17,330 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:17,330 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:17,331 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:17,331 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:17,331 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:17,331 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:17,331 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:17,331 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:17,331 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:17,331 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:17,331 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,331 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:17,332 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:17,332 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/91cab64113484cd7a629985979c5ed24 2023-04-28 14:58:17,332 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,332 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:17,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:17,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:17,333 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:17,333 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:17,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:17,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:17,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:17,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:17,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:17,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c9f82c2[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:17,335 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:17,335 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:17,335 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:17,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:17,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:17,336 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:17,336 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:17,336 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:17,336 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,336 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:17,337 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,337 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:17,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:17,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:17,337 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:17,338 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:17,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:17,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:17,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:17,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:17,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:17,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:17,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:17,340 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:17,340 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:17,340 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:17,340 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:17,340 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:17,341 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:17,341 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:17,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:17,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:17,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:17,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:17,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:17,341 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:17,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:17,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:17,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:17,342 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:17,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:17,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:17,342 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/96f992860b9d4524bf3852f73349996a 2023-04-28 14:58:17,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:17,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,343 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:17,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:17,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,346 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,346 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:17,346 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:17,346 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:17,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:17,346 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:17,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:17,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:17,346 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/b3b7d05686a74d62b2e7e8bd96c4a927 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/b3b7d05686a74d62b2e7e8bd96c4a927 2023-04-28 14:58:17,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:17,346 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:17,346 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:17,346 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,346 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:17,346 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:17,346 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:17,346 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:17,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:17,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:17,349 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/b3b7d05686a74d62b2e7e8bd96c4a927, entries=150, sequenceid=488, filesize=11.9 K 2023-04-28 14:58:17,350 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/91cab64113484cd7a629985979c5ed24 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/91cab64113484cd7a629985979c5ed24 2023-04-28 14:58:17,353 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/91cab64113484cd7a629985979c5ed24, entries=150, sequenceid=488, filesize=11.9 K 2023-04-28 14:58:17,353 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/96f992860b9d4524bf3852f73349996a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/96f992860b9d4524bf3852f73349996a 2023-04-28 14:58:17,358 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/96f992860b9d4524bf3852f73349996a, entries=150, sequenceid=488, filesize=11.9 K 2023-04-28 14:58:17,358 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for f40347916069de574ae857f0751743f8 in 444ms, sequenceid=488, compaction requested=true 2023-04-28 14:58:17,358 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:17,359 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:17,359 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:58:17,359 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:58:17,359 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:17,359 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:17,365 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:58:17,365 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): f40347916069de574ae857f0751743f8/B is initiating minor compaction (all files) 2023-04-28 14:58:17,365 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:58:17,365 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f40347916069de574ae857f0751743f8/B in TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:17,365 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): f40347916069de574ae857f0751743f8/A is initiating minor compaction (all files) 2023-04-28 14:58:17,365 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/7db1ba1de2dd473183e226ad49a8966d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/337426151a5c49f689052d8dbc9df36b, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/91cab64113484cd7a629985979c5ed24] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp, totalSize=36.4 K 2023-04-28 14:58:17,365 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f40347916069de574ae857f0751743f8/A in TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:17,366 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/87d7ca87a0a941e8924b9df1eb907707, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/e3a0bc88f2734554bb553a1594e44c7c, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/b3b7d05686a74d62b2e7e8bd96c4a927] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp, totalSize=36.4 K 2023-04-28 14:58:17,366 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 87d7ca87a0a941e8924b9df1eb907707, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1682693894823 2023-04-28 14:58:17,366 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 7db1ba1de2dd473183e226ad49a8966d, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1682693894823 2023-04-28 14:58:17,366 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting e3a0bc88f2734554bb553a1594e44c7c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1682693895163 2023-04-28 14:58:17,366 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 337426151a5c49f689052d8dbc9df36b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1682693895163 2023-04-28 14:58:17,367 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting b3b7d05686a74d62b2e7e8bd96c4a927, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1682693896301 2023-04-28 14:58:17,367 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 91cab64113484cd7a629985979c5ed24, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1682693896301 2023-04-28 14:58:17,373 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): f40347916069de574ae857f0751743f8#B#compaction#252 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:17,375 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): f40347916069de574ae857f0751743f8#A#compaction#253 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:17,386 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/e5b4664f4d57432eb4843aa729e8af6a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/e5b4664f4d57432eb4843aa729e8af6a 2023-04-28 14:58:17,389 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f40347916069de574ae857f0751743f8/A of f40347916069de574ae857f0751743f8 into e5b4664f4d57432eb4843aa729e8af6a(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:17,389 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:17,389 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8., storeName=f40347916069de574ae857f0751743f8/A, priority=13, startTime=1682693897359; duration=0sec 2023-04-28 14:58:17,390 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:17,390 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:58:17,390 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:58:17,390 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): f40347916069de574ae857f0751743f8/C is initiating minor compaction (all files) 2023-04-28 14:58:17,391 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f40347916069de574ae857f0751743f8/C in TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:17,391 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/6ee09e2ef88a4cf5963ebf6c2fca9edc, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/d3665f247c9a4e53a10b851c670946e2, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/96f992860b9d4524bf3852f73349996a] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp, totalSize=36.4 K 2023-04-28 14:58:17,391 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 6ee09e2ef88a4cf5963ebf6c2fca9edc, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1682693894823 2023-04-28 14:58:17,391 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting d3665f247c9a4e53a10b851c670946e2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1682693895163 2023-04-28 14:58:17,391 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 96f992860b9d4524bf3852f73349996a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1682693896301 2023-04-28 14:58:17,396 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): f40347916069de574ae857f0751743f8#C#compaction#254 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:17,447 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:17,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:17,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:17,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:17,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:17,449 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:17,449 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:17,449 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:17,449 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:17,450 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:17,450 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:17,450 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:17,450 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:17,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:17,451 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,451 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:17,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:17,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:17,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:17,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:17,451 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:17,452 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:17,452 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:17,452 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:17,452 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:17,452 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:17,452 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:17,452 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:17,452 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:17,452 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:17,452 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f40347916069de574ae857f0751743f8 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-28 14:58:17,453 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=A 2023-04-28 14:58:17,453 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:17,453 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=B 2023-04-28 14:58:17,453 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:17,453 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=C 2023-04-28 14:58:17,453 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:17,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on f40347916069de574ae857f0751743f8 2023-04-28 14:58:17,544 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:17,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:17,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50218 deadline: 1682693957556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:17,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50300 deadline: 1682693957558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,660 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:17,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50218 deadline: 1682693957659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,662 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:17,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50300 deadline: 1682693957660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,793 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/61faf19f89124b7aaafdf91ff288f2d4 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/61faf19f89124b7aaafdf91ff288f2d4 2023-04-28 14:58:17,797 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f40347916069de574ae857f0751743f8/B of f40347916069de574ae857f0751743f8 into 61faf19f89124b7aaafdf91ff288f2d4(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:17,797 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:17,797 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8., storeName=f40347916069de574ae857f0751743f8/B, priority=13, startTime=1682693897359; duration=0sec 2023-04-28 14:58:17,797 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:17,811 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/06582355a1cf48688ecd82b6dd274494 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/06582355a1cf48688ecd82b6dd274494 2023-04-28 14:58:17,814 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f40347916069de574ae857f0751743f8/C of f40347916069de574ae857f0751743f8 into 06582355a1cf48688ecd82b6dd274494(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:17,814 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:17,814 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8., storeName=f40347916069de574ae857f0751743f8/C, priority=13, startTime=1682693897359; duration=0sec 2023-04-28 14:58:17,814 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:17,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:17,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50270 deadline: 1682693957858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,860 DEBUG [Thread-1429] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4131 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8., hostname=jenkins-hbase17.apache.org,37181,1682693812482, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 14:58:17,861 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=507 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/3608c8eff05b4550a28c3295bea58b56 2023-04-28 14:58:17,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:17,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693957861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:17,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50218 deadline: 1682693957861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,863 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:17,863 DEBUG [Thread-1431] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4138 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8., hostname=jenkins-hbase17.apache.org,37181,1682693812482, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 14:58:17,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693957862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,864 DEBUG [Thread-1437] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4137 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8., hostname=jenkins-hbase17.apache.org,37181,1682693812482, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 14:58:17,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:17,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50300 deadline: 1682693957864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:17,876 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=507 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/946082e587954e4eaee4b135ba134817 2023-04-28 14:58:18,166 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:18,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50218 deadline: 1682693958165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:18,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:18,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50300 deadline: 1682693958174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:18,287 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=507 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/276d88797e8141dc8afda128852737d5 2023-04-28 14:58:18,290 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/3608c8eff05b4550a28c3295bea58b56 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/3608c8eff05b4550a28c3295bea58b56 2023-04-28 14:58:18,293 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/3608c8eff05b4550a28c3295bea58b56, entries=150, sequenceid=507, filesize=11.9 K 2023-04-28 14:58:18,294 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/946082e587954e4eaee4b135ba134817 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/946082e587954e4eaee4b135ba134817 2023-04-28 14:58:18,297 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/946082e587954e4eaee4b135ba134817, entries=150, sequenceid=507, filesize=11.9 K 2023-04-28 14:58:18,298 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/276d88797e8141dc8afda128852737d5 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/276d88797e8141dc8afda128852737d5 2023-04-28 14:58:18,300 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/276d88797e8141dc8afda128852737d5, entries=150, sequenceid=507, filesize=11.9 K 2023-04-28 14:58:18,301 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for f40347916069de574ae857f0751743f8 in 849ms, sequenceid=507, compaction requested=false 2023-04-28 14:58:18,301 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:18,301 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:18,301 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:18,301 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:18,301 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:18,301 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:18,301 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:18,305 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:18,305 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:18,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:18,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:18,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:18,306 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:18,306 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:18,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:18,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:18,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:18,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:18,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:18,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:18,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39e512c6[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:18,307 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:18,307 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:18,307 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:18,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:18,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:18,308 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:18,308 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:18,308 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:18,308 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:18,308 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:18,308 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:18,308 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:18,308 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:18,308 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:18,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:18,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:18,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:18,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:18,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:18,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:18,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:18,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:18,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:18,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:18,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:18,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:18,310 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:18,310 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:18,310 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:18,310 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:18,310 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:18,311 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:18,311 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:18,311 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:18,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:18,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:18,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:18,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:18,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:18,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:18,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:18,312 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:18,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:18,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:18,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:18,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:18,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:18,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:18,313 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:18,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:18,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:18,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:18,314 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:18,314 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:18,314 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:18,314 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:18,314 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:18,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:18,314 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:18,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:18,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:18,314 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:18,314 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:18,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:18,314 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:18,314 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:18,314 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:18,314 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:18,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:18,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:18,415 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:18,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:18,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:18,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:18,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:18,417 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:18,417 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:18,417 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:18,417 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:18,420 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:18,420 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:18,422 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:18,422 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:18,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:18,423 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:18,423 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:18,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:18,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:18,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:18,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:18,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:18,433 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:18,433 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:18,433 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:18,433 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:18,433 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:18,433 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:18,433 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:18,433 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:18,433 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:18,433 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f40347916069de574ae857f0751743f8 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-28 14:58:18,434 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=A 2023-04-28 14:58:18,434 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:18,434 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=B 2023-04-28 14:58:18,434 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:18,434 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=C 2023-04-28 14:58:18,434 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:18,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on f40347916069de574ae857f0751743f8 2023-04-28 14:58:18,672 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:18,700 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:18,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50300 deadline: 1682693958699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:18,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:18,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50218 deadline: 1682693958699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:18,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:18,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50300 deadline: 1682693958801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:18,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:18,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50218 deadline: 1682693958801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:18,840 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=523 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/0a2ead3b015b416580d4c9af8a6fca71 2023-04-28 14:58:19,005 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:19,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50300 deadline: 1682693959004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,005 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:19,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50218 deadline: 1682693959005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,250 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=523 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/6d14313cb58b48f9a50d27699b92c251 2023-04-28 14:58:19,258 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=523 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/19c868be962d48e7a6c0412441c94c2f 2023-04-28 14:58:19,261 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/0a2ead3b015b416580d4c9af8a6fca71 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/0a2ead3b015b416580d4c9af8a6fca71 2023-04-28 14:58:19,264 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/0a2ead3b015b416580d4c9af8a6fca71, entries=150, sequenceid=523, filesize=11.9 K 2023-04-28 14:58:19,265 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/6d14313cb58b48f9a50d27699b92c251 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/6d14313cb58b48f9a50d27699b92c251 2023-04-28 14:58:19,268 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/6d14313cb58b48f9a50d27699b92c251, entries=150, sequenceid=523, filesize=11.9 K 2023-04-28 14:58:19,269 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/19c868be962d48e7a6c0412441c94c2f as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/19c868be962d48e7a6c0412441c94c2f 2023-04-28 14:58:19,272 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/19c868be962d48e7a6c0412441c94c2f, entries=150, sequenceid=523, filesize=11.9 K 2023-04-28 14:58:19,272 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for f40347916069de574ae857f0751743f8 in 839ms, sequenceid=523, compaction requested=true 2023-04-28 14:58:19,272 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:19,272 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:19,273 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:19,273 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:19,273 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,273 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:19,273 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:19,274 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,274 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,274 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:19,274 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:19,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:19,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:19,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:19,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:19,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:19,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49888362[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:19,275 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:19,275 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,276 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,276 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:19,276 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:19,276 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:19,276 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,276 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:19,277 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:19,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:19,277 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:19,277 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:19,277 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:19,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:19,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:19,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:19,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:19,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:19,278 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:19,278 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:19,278 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:19,279 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:19,279 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:19,279 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:19,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:19,280 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,280 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:19,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:19,280 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:19,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:19,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:19,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,281 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:19,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,282 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,282 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,282 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:19,282 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:19,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:19,282 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:19,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:19,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:19,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:19,282 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:19,282 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,283 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,283 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:19,283 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,283 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:19,283 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:19,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:19,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on f40347916069de574ae857f0751743f8 2023-04-28 14:58:19,309 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f40347916069de574ae857f0751743f8 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-28 14:58:19,310 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=A 2023-04-28 14:58:19,310 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:19,310 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=B 2023-04-28 14:58:19,310 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:19,310 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=C 2023-04-28 14:58:19,310 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:19,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:19,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50300 deadline: 1682693959321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:19,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50218 deadline: 1682693959323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,383 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:19,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:19,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:19,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:19,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:19,386 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:19,386 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:19,386 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:19,386 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:19,386 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,386 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,387 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:19,387 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:19,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:19,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,387 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,387 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:19,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:19,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:19,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:19,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:19,388 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:19,388 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:19,388 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:19,388 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:19,388 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:19,388 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:19,388 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:19,388 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:19,389 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:19,389 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:19,389 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:19,389 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,389 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:19,389 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:19,389 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,389 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:19,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:19,390 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,390 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:19,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:19,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:19,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:19,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:19,391 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:19,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66d89c3d[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:19,391 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,391 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,392 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:19,392 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:19,392 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,392 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:19,392 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:19,392 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,392 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:19,392 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:19,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:19,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:19,392 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:19,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:19,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:19,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:19,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:19,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:19,395 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:19,395 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:19,395 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:19,395 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:19,395 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:19,396 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,396 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:19,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,396 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:19,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:19,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:19,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:19,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:19,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,397 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:19,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,398 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:19,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,400 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,400 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:19,400 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:19,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:19,400 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:19,400 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,400 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,400 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,400 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,400 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:19,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:19,400 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:19,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:19,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:19,400 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:19,401 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:19,425 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:19,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50300 deadline: 1682693959424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:19,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50218 deadline: 1682693959425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,501 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:19,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:19,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:19,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:19,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:19,503 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:19,503 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:19,503 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:19,503 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:19,504 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,504 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,504 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,504 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:19,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:19,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:19,505 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,505 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:19,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:19,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:19,505 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:19,506 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:19,506 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:19,506 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:19,506 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:19,506 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:19,506 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:19,506 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:19,506 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:19,506 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:19,506 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:19,506 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:19,506 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:19,506 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,506 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:19,506 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:19,507 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,507 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:19,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:19,508 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:19,508 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:19,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:19,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:19,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:19,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76419bb6[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:19,509 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:19,509 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,509 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,510 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,510 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:19,510 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:19,510 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:19,510 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:19,510 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,510 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:19,510 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:19,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:19,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:19,510 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:19,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:19,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:19,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:19,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:19,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:19,512 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:19,512 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:19,512 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:19,512 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:19,512 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:19,513 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,513 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:19,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:19,513 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:19,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:19,513 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:19,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:19,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:19,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,514 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:19,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,516 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:19,516 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:19,516 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:19,516 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:19,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:19,516 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:19,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:19,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:19,516 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,516 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,516 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,516 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,516 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:19,516 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:19,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:19,516 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:19,617 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:19,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:19,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:19,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:19,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:19,619 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:19,619 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:19,619 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:19,619 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:19,620 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,620 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,620 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,620 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:19,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:19,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:19,620 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,620 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:19,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:19,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:19,621 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:19,621 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:19,621 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:19,621 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:19,621 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:19,622 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:19,622 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:19,622 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:19,622 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:19,622 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:19,622 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:19,622 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:19,622 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:19,622 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,622 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:19,622 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:19,623 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,623 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:19,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:19,623 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,623 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:19,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:19,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:19,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:19,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:19,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6723d49a[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:19,624 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:19,624 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,624 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,625 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:19,625 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:19,625 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,625 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:19,625 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:19,625 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,625 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:19,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:19,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:19,625 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:19,626 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:19,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:19,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:19,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:19,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:19,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:19,628 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:19,628 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:19,628 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:19,628 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:19,628 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:19,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:19,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50300 deadline: 1682693959627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,629 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,629 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:19,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,629 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:19,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:19,629 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:19,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,630 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:19,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:19,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50218 deadline: 1682693959629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,631 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,631 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:19,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:19,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:19,631 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,632 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:19,632 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:19,632 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:19,632 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:19,632 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,632 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,632 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,632 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:19,632 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:19,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:19,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:19,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:19,717 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=543 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/c552d48303514707a036ea1c2b0cb83d 2023-04-28 14:58:19,732 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:19,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:19,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:19,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:19,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:19,735 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:19,735 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:19,735 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:19,735 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:19,735 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,735 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,736 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:19,736 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:19,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:19,736 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,736 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:19,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:19,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,737 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:19,737 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:19,737 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:19,737 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:19,737 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:19,737 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:19,737 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:19,737 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:19,737 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:19,737 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:19,737 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:19,737 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:19,737 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:19,737 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:19,737 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,738 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:19,738 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:19,738 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,738 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:19,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:19,739 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,739 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:19,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:19,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:19,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:19,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:19,740 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:19,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a3dc88f[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:19,740 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,740 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,740 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:19,740 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:19,741 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:19,741 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,741 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:19,741 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:19,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:19,741 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:19,741 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:19,741 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:19,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:19,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:19,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:19,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:19,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:19,743 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:19,743 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:19,743 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:19,743 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:19,743 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:19,744 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,744 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:19,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,744 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:19,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:19,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:19,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:19,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:19,745 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:19,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,746 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:19,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,747 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:19,747 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:19,747 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,747 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,747 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,747 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,747 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,747 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,747 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,747 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:19,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:19,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:19,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:19,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:19,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:19,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:19,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:19,749 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:19,749 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:19,850 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:19,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:19,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:19,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:19,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:19,852 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:19,852 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:19,852 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:19,852 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:19,853 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,853 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,853 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,853 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:19,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:19,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:19,854 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,854 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:19,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:19,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,854 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:19,854 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:19,854 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:19,855 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:19,855 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:19,855 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:19,855 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:19,855 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:19,855 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:19,855 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:19,855 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:19,855 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:19,855 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:19,855 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:19,855 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,855 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:19,855 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:19,856 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,856 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:19,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:19,856 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,856 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:19,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:19,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:19,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:19,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:19,857 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:19,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48268881[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:19,857 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,858 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,858 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:19,858 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,858 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:19,858 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:19,858 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:19,859 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:19,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:19,859 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:19,859 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:19,859 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:19,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:19,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:19,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:19,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:19,861 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:19,861 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:19,861 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:19,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:19,861 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:19,861 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:19,862 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,862 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:19,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:19,862 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:19,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:19,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:19,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:19,862 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:19,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,863 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:19,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,864 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,864 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:19,864 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,864 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:19,864 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:19,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:19,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:19,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:19,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:19,864 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,865 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,865 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,865 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,865 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,865 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:19,865 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:19,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:19,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:19,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:19,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50300 deadline: 1682693959931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:19,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50218 deadline: 1682693959932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,965 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:19,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:19,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:19,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:19,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:19,968 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:19,968 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:19,968 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:19,968 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:19,968 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,968 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,969 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,969 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:19,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:19,969 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,969 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:19,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,970 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:19,970 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:19,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:19,970 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:19,970 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:19,970 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:19,970 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:19,970 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:19,970 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:19,970 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:19,970 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:19,970 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:19,970 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:19,970 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:19,971 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,971 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:19,971 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:19,971 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,971 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:19,971 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,971 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:19,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:19,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32403618[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:19,973 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:19,973 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,973 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,973 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:19,973 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:19,973 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,973 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:19,973 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:19,974 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:19,974 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:19,974 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,974 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:19,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:19,976 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:19,976 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:19,976 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:19,976 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:19,976 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:19,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:19,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:19,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:19,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:19,977 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:19,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:19,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:19,978 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:19,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:19,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,980 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:19,980 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,980 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:19,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:19,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:19,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:19,980 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:19,980 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,980 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:19,980 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:19,980 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,980 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:19,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:19,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:19,980 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:19,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:19,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:19,981 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:19,981 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:20,081 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:20,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:20,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:20,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:20,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:20,083 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:20,083 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:20,083 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:20,083 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:20,084 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:20,084 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:20,085 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:20,085 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:20,085 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,085 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:20,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:20,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:20,086 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:20,086 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:20,086 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:20,086 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:20,086 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:20,086 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:20,086 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:20,086 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:20,086 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:20,086 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:20,086 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:20,086 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:20,086 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:20,087 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,087 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:20,087 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:20,087 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,087 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:20,088 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:20,088 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:20,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:20,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b3ea3ce[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:20,089 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:20,089 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:20,090 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:20,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:20,090 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:20,090 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:20,090 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:20,090 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,090 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:20,091 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,091 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:20,091 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:20,091 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:20,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:20,094 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:20,094 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:20,094 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:20,094 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:20,094 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:20,095 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:20,095 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:20,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:20,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:20,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:20,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:20,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:20,095 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:20,096 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:20,097 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:20,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:20,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:20,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:20,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:20,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:20,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:20,099 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:20,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:20,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:20,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:20,099 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:20,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:20,100 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:20,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:20,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:20,125 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=543 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/76a89d2af30b44db928b9d761556566d 2023-04-28 14:58:20,147 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=543 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/fba04e031cb0455fbbf4dadb3e849d70 2023-04-28 14:58:20,152 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/c552d48303514707a036ea1c2b0cb83d as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/c552d48303514707a036ea1c2b0cb83d 2023-04-28 14:58:20,155 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/c552d48303514707a036ea1c2b0cb83d, entries=150, sequenceid=543, filesize=11.9 K 2023-04-28 14:58:20,156 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/76a89d2af30b44db928b9d761556566d as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/76a89d2af30b44db928b9d761556566d 2023-04-28 14:58:20,159 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/76a89d2af30b44db928b9d761556566d, entries=150, sequenceid=543, filesize=11.9 K 2023-04-28 14:58:20,160 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/fba04e031cb0455fbbf4dadb3e849d70 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/fba04e031cb0455fbbf4dadb3e849d70 2023-04-28 14:58:20,163 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/fba04e031cb0455fbbf4dadb3e849d70, entries=150, sequenceid=543, filesize=11.9 K 2023-04-28 14:58:20,164 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for f40347916069de574ae857f0751743f8 in 855ms, sequenceid=543, compaction requested=true 2023-04-28 14:58:20,164 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:20,164 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:20,164 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:58:20,164 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:58:20,164 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:20,164 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:20,165 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:58:20,165 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:58:20,165 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): f40347916069de574ae857f0751743f8/A is initiating minor compaction (all files) 2023-04-28 14:58:20,165 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): f40347916069de574ae857f0751743f8/B is initiating minor compaction (all files) 2023-04-28 14:58:20,165 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f40347916069de574ae857f0751743f8/A in TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:20,165 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f40347916069de574ae857f0751743f8/B in TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:20,165 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/e5b4664f4d57432eb4843aa729e8af6a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/3608c8eff05b4550a28c3295bea58b56, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/0a2ead3b015b416580d4c9af8a6fca71, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/c552d48303514707a036ea1c2b0cb83d] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp, totalSize=48.4 K 2023-04-28 14:58:20,165 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/61faf19f89124b7aaafdf91ff288f2d4, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/946082e587954e4eaee4b135ba134817, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/6d14313cb58b48f9a50d27699b92c251, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/76a89d2af30b44db928b9d761556566d] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp, totalSize=48.4 K 2023-04-28 14:58:20,165 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting e5b4664f4d57432eb4843aa729e8af6a, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1682693896301 2023-04-28 14:58:20,165 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 61faf19f89124b7aaafdf91ff288f2d4, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1682693896301 2023-04-28 14:58:20,166 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 3608c8eff05b4550a28c3295bea58b56, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1682693896927 2023-04-28 14:58:20,166 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 946082e587954e4eaee4b135ba134817, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1682693896927 2023-04-28 14:58:20,166 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 0a2ead3b015b416580d4c9af8a6fca71, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1682693897550 2023-04-28 14:58:20,166 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 6d14313cb58b48f9a50d27699b92c251, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1682693897550 2023-04-28 14:58:20,167 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 76a89d2af30b44db928b9d761556566d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=543, earliestPutTs=1682693898697 2023-04-28 14:58:20,167 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting c552d48303514707a036ea1c2b0cb83d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=543, earliestPutTs=1682693898697 2023-04-28 14:58:20,172 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): f40347916069de574ae857f0751743f8#B#compaction#265 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:20,172 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): f40347916069de574ae857f0751743f8#A#compaction#264 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:20,183 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/44775bd721ed4269b21562b504523567 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/44775bd721ed4269b21562b504523567 2023-04-28 14:58:20,187 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f40347916069de574ae857f0751743f8/B of f40347916069de574ae857f0751743f8 into 44775bd721ed4269b21562b504523567(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:20,187 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:20,187 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8., storeName=f40347916069de574ae857f0751743f8/B, priority=12, startTime=1682693900164; duration=0sec 2023-04-28 14:58:20,187 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:20,187 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:58:20,188 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:58:20,188 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): f40347916069de574ae857f0751743f8/C is initiating minor compaction (all files) 2023-04-28 14:58:20,188 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f40347916069de574ae857f0751743f8/C in TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:20,188 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/06582355a1cf48688ecd82b6dd274494, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/276d88797e8141dc8afda128852737d5, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/19c868be962d48e7a6c0412441c94c2f, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/fba04e031cb0455fbbf4dadb3e849d70] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp, totalSize=48.4 K 2023-04-28 14:58:20,189 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 06582355a1cf48688ecd82b6dd274494, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1682693896301 2023-04-28 14:58:20,189 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 276d88797e8141dc8afda128852737d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1682693896927 2023-04-28 14:58:20,189 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 19c868be962d48e7a6c0412441c94c2f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1682693897550 2023-04-28 14:58:20,189 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting fba04e031cb0455fbbf4dadb3e849d70, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=543, earliestPutTs=1682693898697 2023-04-28 14:58:20,194 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): f40347916069de574ae857f0751743f8#C#compaction#266 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:20,200 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:20,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:20,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:20,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:20,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:20,206 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:20,206 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:20,206 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:20,206 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:20,206 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:20,206 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:20,207 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:20,207 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:20,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:20,207 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,208 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:20,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:20,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:20,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:20,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:20,208 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:20,208 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:20,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:20,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:20,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:20,209 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:20,209 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:20,209 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:20,209 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:20,209 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:20,209 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f40347916069de574ae857f0751743f8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-28 14:58:20,209 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=A 2023-04-28 14:58:20,209 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:20,209 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=B 2023-04-28 14:58:20,209 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:20,209 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=C 2023-04-28 14:58:20,209 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:20,210 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/876da018e8ff4b68b2164bea8e3869e0 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/876da018e8ff4b68b2164bea8e3869e0 2023-04-28 14:58:20,213 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f40347916069de574ae857f0751743f8/C of f40347916069de574ae857f0751743f8 into 876da018e8ff4b68b2164bea8e3869e0(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:20,213 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:20,213 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8., storeName=f40347916069de574ae857f0751743f8/C, priority=12, startTime=1682693900164; duration=0sec 2023-04-28 14:58:20,214 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:20,215 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=556 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/7f0dd84c7caf4d008daae4eeae4ba4de 2023-04-28 14:58:20,224 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=556 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/5de777c730b545d0af8b98e937c16d38 2023-04-28 14:58:20,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on f40347916069de574ae857f0751743f8 2023-04-28 14:58:20,437 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:20,461 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:20,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50218 deadline: 1682693960459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,461 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:20,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50300 deadline: 1682693960460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,563 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:20,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50218 deadline: 1682693960562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,564 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:20,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50300 deadline: 1682693960562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,579 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/857bb5128b7645c59f2b16ac34a79b9c as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/857bb5128b7645c59f2b16ac34a79b9c 2023-04-28 14:58:20,582 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f40347916069de574ae857f0751743f8/A of f40347916069de574ae857f0751743f8 into 857bb5128b7645c59f2b16ac34a79b9c(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:20,582 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:20,582 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8., storeName=f40347916069de574ae857f0751743f8/A, priority=12, startTime=1682693900164; duration=0sec 2023-04-28 14:58:20,582 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:20,636 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=556 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/612acf70959d47aebf41bae37f083329 2023-04-28 14:58:20,639 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/7f0dd84c7caf4d008daae4eeae4ba4de as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/7f0dd84c7caf4d008daae4eeae4ba4de 2023-04-28 14:58:20,642 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/7f0dd84c7caf4d008daae4eeae4ba4de, entries=150, sequenceid=556, filesize=11.9 K 2023-04-28 14:58:20,643 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/5de777c730b545d0af8b98e937c16d38 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/5de777c730b545d0af8b98e937c16d38 2023-04-28 14:58:20,646 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/5de777c730b545d0af8b98e937c16d38, entries=150, sequenceid=556, filesize=11.9 K 2023-04-28 14:58:20,646 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/612acf70959d47aebf41bae37f083329 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/612acf70959d47aebf41bae37f083329 2023-04-28 14:58:20,649 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/612acf70959d47aebf41bae37f083329, entries=150, sequenceid=556, filesize=11.9 K 2023-04-28 14:58:20,650 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for f40347916069de574ae857f0751743f8 in 441ms, sequenceid=556, compaction requested=false 2023-04-28 14:58:20,650 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:20,650 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:20,650 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:20,650 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:20,650 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,650 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:20,650 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:20,651 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:20,651 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:20,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:20,651 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:20,651 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:20,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:20,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:20,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:20,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:20,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:20,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fc36396[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:20,652 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:20,652 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:20,653 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:20,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:20,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:20,653 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:20,653 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,653 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:20,653 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:20,653 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:20,653 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,653 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:20,654 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:20,654 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:20,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:20,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:20,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:20,655 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:20,655 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:20,655 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:20,655 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:20,655 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:20,656 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:20,656 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:20,656 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:20,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:20,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:20,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:20,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:20,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:20,657 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:20,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:20,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:20,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:20,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:20,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:20,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:20,658 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,659 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,659 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:20,659 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:20,659 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:20,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:20,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:20,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:20,659 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:20,659 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:20,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:20,659 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:20,659 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,659 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:20,660 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:20,660 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:20,659 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:20,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:20,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:20,759 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:20,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:20,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:20,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:20,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:20,762 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:20,762 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:20,762 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:20,762 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:20,763 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:20,763 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:20,763 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,763 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:20,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:20,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:20,763 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,763 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:20,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:20,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:20,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:20,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:20,764 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:20,764 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:20,764 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:20,765 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:20,765 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:20,765 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:20,765 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:20,765 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f40347916069de574ae857f0751743f8 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-28 14:58:20,765 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=A 2023-04-28 14:58:20,765 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:20,765 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=B 2023-04-28 14:58:20,765 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:20,765 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=C 2023-04-28 14:58:20,765 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:20,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on f40347916069de574ae857f0751743f8 2023-04-28 14:58:20,768 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:20,774 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/439a180c6db24b3db50091acca8fa838 2023-04-28 14:58:20,779 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:20,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50300 deadline: 1682693960777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:20,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50218 deadline: 1682693960778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,881 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:20,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50300 deadline: 1682693960880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:20,881 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:20,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50218 deadline: 1682693960881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:21,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:21,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50300 deadline: 1682693961082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:21,084 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:21,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50218 deadline: 1682693961082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:21,183 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/aae4423866f54539a73941137f501a6a 2023-04-28 14:58:21,386 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:21,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50218 deadline: 1682693961384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:21,387 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:21,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50300 deadline: 1682693961385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:21,593 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/50f2348452114b26a98ff5f3b5004ab6 2023-04-28 14:58:21,596 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/439a180c6db24b3db50091acca8fa838 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/439a180c6db24b3db50091acca8fa838 2023-04-28 14:58:21,599 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/439a180c6db24b3db50091acca8fa838, entries=150, sequenceid=578, filesize=11.9 K 2023-04-28 14:58:21,600 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/aae4423866f54539a73941137f501a6a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/aae4423866f54539a73941137f501a6a 2023-04-28 14:58:21,603 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/aae4423866f54539a73941137f501a6a, entries=150, sequenceid=578, filesize=11.9 K 2023-04-28 14:58:21,603 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/50f2348452114b26a98ff5f3b5004ab6 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/50f2348452114b26a98ff5f3b5004ab6 2023-04-28 14:58:21,613 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/50f2348452114b26a98ff5f3b5004ab6, entries=150, sequenceid=578, filesize=11.9 K 2023-04-28 14:58:21,613 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for f40347916069de574ae857f0751743f8 in 848ms, sequenceid=578, compaction requested=true 2023-04-28 14:58:21,613 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:21,613 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:21,613 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:21,614 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:21,614 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:21,614 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:21,614 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:21,614 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:21,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:21,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:21,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:21,614 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:21,615 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:21,615 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:21,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:21,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:21,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:21,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:21,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:21,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:21,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33dd986f[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:21,616 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:21,616 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:21,617 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:21,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:21,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:21,617 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:21,617 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:21,617 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:21,617 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:21,617 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:21,618 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:21,618 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:21,618 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:21,618 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:21,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:21,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:21,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:21,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:21,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:21,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:21,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:21,620 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:21,620 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:21,620 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:21,620 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:21,620 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:21,621 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:21,621 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:21,621 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:21,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:21,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:21,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:21,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:21,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:21,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:21,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:21,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:21,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:21,622 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:21,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:21,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:21,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:21,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:21,623 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:21,624 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:21,624 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:21,624 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:21,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:21,624 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:21,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:21,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:21,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:21,625 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:21,624 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:21,625 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:21,625 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:21,625 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:21,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:21,625 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:21,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:21,625 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:21,625 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:21,725 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:21,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:21,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:21,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:21,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:21,728 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:21,728 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:21,728 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:21,728 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:21,728 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:21,728 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:21,729 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:21,729 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:21,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:21,729 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:21,729 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:21,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:21,729 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:21,730 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:21,730 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:21,730 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:21,730 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:21,730 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:21,730 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:21,730 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:21,730 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:21,730 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. started... 2023-04-28 14:58:21,730 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f40347916069de574ae857f0751743f8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-28 14:58:21,730 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=A 2023-04-28 14:58:21,730 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:21,730 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=B 2023-04-28 14:58:21,730 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:21,730 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=C 2023-04-28 14:58:21,730 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:21,754 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/67ba696503a24251a4e79d7c345d6320 2023-04-28 14:58:21,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on f40347916069de574ae857f0751743f8 2023-04-28 14:58:21,884 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. as already flushing 2023-04-28 14:58:21,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:21,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693961899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:21,904 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:21,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50218 deadline: 1682693961900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:21,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:21,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693961901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:21,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:21,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50270 deadline: 1682693961901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:21,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:21,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50300 deadline: 1682693961903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:22,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:22,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50260 deadline: 1682693962004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:22,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:22,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50218 deadline: 1682693962004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:22,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:22,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50270 deadline: 1682693962005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:22,008 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:22,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:50300 deadline: 1682693962007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:22,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:22,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:50244 deadline: 1682693962007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:22,077 DEBUG [Thread-1444] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d22a55e to 127.0.0.1:61762 2023-04-28 14:58:22,077 DEBUG [Thread-1444] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:58:22,078 DEBUG [Thread-1442] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3c5d1710 to 127.0.0.1:61762 2023-04-28 14:58:22,078 DEBUG [Thread-1442] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:58:22,078 DEBUG [Thread-1448] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7897b06a to 127.0.0.1:61762 2023-04-28 14:58:22,078 DEBUG [Thread-1448] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:58:22,079 DEBUG [Thread-1446] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2179ead9 to 127.0.0.1:61762 2023-04-28 14:58:22,079 DEBUG [Thread-1446] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:58:22,080 DEBUG [Thread-1440] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4cbc74e7 to 127.0.0.1:61762 2023-04-28 14:58:22,080 DEBUG [Thread-1440] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:58:22,162 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/e347f0d08a0a411fb6f18a957a36fb89 2023-04-28 14:58:22,180 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/b7774ace84224ae7b4e3d123bb0d3200 2023-04-28 14:58:22,184 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/67ba696503a24251a4e79d7c345d6320 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/67ba696503a24251a4e79d7c345d6320 2023-04-28 14:58:22,189 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/67ba696503a24251a4e79d7c345d6320, entries=150, sequenceid=590, filesize=11.9 K 2023-04-28 14:58:22,190 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/e347f0d08a0a411fb6f18a957a36fb89 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/e347f0d08a0a411fb6f18a957a36fb89 2023-04-28 14:58:22,194 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/e347f0d08a0a411fb6f18a957a36fb89, entries=150, sequenceid=590, filesize=11.9 K 2023-04-28 14:58:22,196 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/b7774ace84224ae7b4e3d123bb0d3200 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/b7774ace84224ae7b4e3d123bb0d3200 2023-04-28 14:58:22,201 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/b7774ace84224ae7b4e3d123bb0d3200, entries=150, sequenceid=590, filesize=11.9 K 2023-04-28 14:58:22,202 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for f40347916069de574ae857f0751743f8 in 472ms, sequenceid=590, compaction requested=true 2023-04-28 14:58:22,202 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:22,202 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:22,202 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:22,202 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:22,202 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:22,202 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:22,202 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:22,204 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:22,204 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:22,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:22,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:22,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:22,204 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:22,204 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:22,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:22,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:22,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:22,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:22,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:22,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:22,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9e88fa[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:22,206 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:22,207 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:22,207 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:22,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:22,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:22,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:22,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:22,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:22,208 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:22,208 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:22,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on f40347916069de574ae857f0751743f8 2023-04-28 14:58:22,209 DEBUG [Thread-1431] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3f837278 to 127.0.0.1:61762 2023-04-28 14:58:22,209 DEBUG [Thread-1431] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:58:22,209 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f40347916069de574ae857f0751743f8 3/3 column families, dataSize=127.47 KB heapSize=379.97 KB 2023-04-28 14:58:22,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693901896, max timestamp=1682693902207Num uniques 859; 2023-04-28 14:58:22,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:58:22,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693901896, max timestamp=1682693902207Num uniques 859; 2023-04-28 14:58:22,210 DEBUG [Thread-1435] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6347f2e6 to 127.0.0.1:61762 2023-04-28 14:58:22,210 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=A 2023-04-28 14:58:22,210 DEBUG [Thread-1435] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:58:22,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:58:22,210 DEBUG [Thread-1429] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x46d74611 to 127.0.0.1:61762 2023-04-28 14:58:22,211 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:22,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693901896, max timestamp=1682693902207Num uniques 859; 2023-04-28 14:58:22,211 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=B 2023-04-28 14:58:22,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:58:22,211 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:22,211 DEBUG [Thread-1429] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:58:22,212 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=C 2023-04-28 14:58:22,212 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:22,212 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:22,212 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:22,212 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:22,212 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:22,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:22,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:22,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:22,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:22,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:22,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:22,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:22,214 DEBUG [Thread-1437] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x36523181 to 127.0.0.1:61762 2023-04-28 14:58:22,214 DEBUG [Thread-1433] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a8a575d to 127.0.0.1:61762 2023-04-28 14:58:22,214 DEBUG [Thread-1437] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:58:22,214 DEBUG [Thread-1433] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:58:22,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:22,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:22,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:22,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:22,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:22,216 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:22,216 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:22,216 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:22,216 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:22,216 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:22,217 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:22,217 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:22,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:22,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:22,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:22,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:22,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:22,217 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:22,218 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:22,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:22,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:22,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:22,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:22,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:22,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:22,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:22,219 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:22,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:22,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:22,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:22,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:22,221 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:22,221 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:22,221 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:22,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:22,221 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:22,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:22,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:22,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:22,221 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:22,221 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:22,221 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:22,221 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:22,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:22,221 DEBUG [Thread-1439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:22,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:22,221 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:22,221 DEBUG [Thread-1439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:22,221 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:22,321 DEBUG [Thread-1439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:22,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:22,324 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-28 14:58:22,324 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(394): wrote 251 2023-04-28 14:58:22,324 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(394): wrote 36 2023-04-28 14:58:22,324 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(394): wrote 94 2023-04-28 14:58:22,324 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(394): wrote 93 2023-04-28 14:58:22,324 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(394): wrote 40 2023-04-28 14:58:22,324 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-28 14:58:22,325 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(398): read 6356 2023-04-28 14:58:22,325 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(398): read 6040 2023-04-28 14:58:22,325 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(398): read 6456 2023-04-28 14:58:22,325 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(398): read 6332 2023-04-28 14:58:22,325 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(398): read 6070 2023-04-28 14:58:22,325 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-28 14:58:22,325 INFO [Listener at localhost.localdomain/40193] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-28 14:58:22,326 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c466b6c to 127.0.0.1:61762 2023-04-28 14:58:22,326 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:58:22,327 INFO [Listener at localhost.localdomain/40193] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-28 14:58:22,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.HMaster$11(2418): Client=jenkins//136.243.18.41 disable TestAcidGuarantees 2023-04-28 14:58:22,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-28 14:58:22,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-28 14:58:22,335 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682693902335"}]},"ts":"1682693902335"} 2023-04-28 14:58:22,337 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-28 14:58:22,339 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-28 14:58:22,340 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f40347916069de574ae857f0751743f8, UNASSIGN}] 2023-04-28 14:58:22,342 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f40347916069de574ae857f0751743f8, UNASSIGN 2023-04-28 14:58:22,343 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=f40347916069de574ae857f0751743f8, regionState=CLOSING, regionLocation=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:22,343 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693902343"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682693902343"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682693902343"}]},"ts":"1682693902343"} 2023-04-28 14:58:22,345 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482}] 2023-04-28 14:58:22,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-28 14:58:22,497 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(109): Close f40347916069de574ae857f0751743f8 2023-04-28 14:58:22,498 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1604): Closing f40347916069de574ae857f0751743f8, disabling compactions & flushes 2023-04-28 14:58:22,498 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:22,621 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=613 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/488fb78be30d49a88e374233c562547b 2023-04-28 14:58:22,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-28 14:58:22,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-28 14:58:23,040 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=613 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/26ffc1fb8107424db01556d0da074841 2023-04-28 14:58:23,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-28 14:58:23,458 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=613 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/2661ac2e3e62452db8c0d30643b53837 2023-04-28 14:58:23,465 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/488fb78be30d49a88e374233c562547b as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/488fb78be30d49a88e374233c562547b 2023-04-28 14:58:23,468 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/488fb78be30d49a88e374233c562547b, entries=150, sequenceid=613, filesize=11.9 K 2023-04-28 14:58:23,468 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/26ffc1fb8107424db01556d0da074841 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/26ffc1fb8107424db01556d0da074841 2023-04-28 14:58:23,471 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/26ffc1fb8107424db01556d0da074841, entries=150, sequenceid=613, filesize=11.9 K 2023-04-28 14:58:23,471 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/2661ac2e3e62452db8c0d30643b53837 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/2661ac2e3e62452db8c0d30643b53837 2023-04-28 14:58:23,474 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/2661ac2e3e62452db8c0d30643b53837, entries=150, sequenceid=613, filesize=11.9 K 2023-04-28 14:58:23,475 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~106.03 KB/108576, currentSize=13.42 KB/13740 for f40347916069de574ae857f0751743f8 in 1266ms, sequenceid=613, compaction requested=true 2023-04-28 14:58:23,475 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:23,475 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:23,475 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:23,475 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. because compaction request was cancelled 2023-04-28 14:58:23,475 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. because compaction request was cancelled 2023-04-28 14:58:23,475 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:23,475 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. after waiting 0 ms 2023-04-28 14:58:23,475 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:23,475 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:23,475 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2745): Flushing f40347916069de574ae857f0751743f8 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-28 14:58:23,475 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. because compaction request was cancelled 2023-04-28 14:58:23,475 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:23,475 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=A 2023-04-28 14:58:23,475 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:23,475 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=B 2023-04-28 14:58:23,475 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:23,475 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f40347916069de574ae857f0751743f8, store=C 2023-04-28 14:58:23,475 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:23,881 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=618 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/5cc6ef6e25b34841acc22852092f57ff 2023-04-28 14:58:24,299 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=618 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/7a803a4346d349739a699732238fbe88 2023-04-28 14:58:24,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-28 14:58:24,709 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=618 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/747c0bad303f47f5800307f7a017e1a7 2023-04-28 14:58:24,721 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/A/5cc6ef6e25b34841acc22852092f57ff as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/5cc6ef6e25b34841acc22852092f57ff 2023-04-28 14:58:24,728 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/5cc6ef6e25b34841acc22852092f57ff, entries=100, sequenceid=618, filesize=9.5 K 2023-04-28 14:58:24,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/B/7a803a4346d349739a699732238fbe88 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/7a803a4346d349739a699732238fbe88 2023-04-28 14:58:24,735 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/7a803a4346d349739a699732238fbe88, entries=100, sequenceid=618, filesize=9.5 K 2023-04-28 14:58:24,736 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/.tmp/C/747c0bad303f47f5800307f7a017e1a7 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/747c0bad303f47f5800307f7a017e1a7 2023-04-28 14:58:24,742 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/747c0bad303f47f5800307f7a017e1a7, entries=100, sequenceid=618, filesize=9.5 K 2023-04-28 14:58:24,743 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=0 B/0 for f40347916069de574ae857f0751743f8 in 1268ms, sequenceid=618, compaction requested=true 2023-04-28 14:58:24,763 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/70d377b4e88a4cc3adbd1d7b18ba121a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/c9c6d4e6f7ba4c62ad47067533b6baf6, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/f156cfd423824719b685280a380f831a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/7b5b284ced40412681ecd698d1098760, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/94d8d51e11534d59bf433357f9f830f1, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/64aa34bfcbea4485a9f771c9860e6cef, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/63bc9271a4ee4fe286cf9f7cefef0483, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/ac2ecd90132941448718a1603cff8c42, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/7a27d40375534ea5aba4d349aac90c32, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/d2627b4b297c481b8459a87a58bbfd99, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/e2098ac32c3c4aabb78f456a864e2418, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/15876e573f484482bc1ae855b967981b, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/651de671b37e448db714e54c51162f6d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/5f3b4fd07a2a4f89bcc94c29c5858265, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/e5c0df22792a413b8dda6341feaafd5a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/7f125540dcdd4e06ada696b564098d06, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/a6fcc77810a64359ade5b8283483cdb1, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/664c55fb574042ca9bf017a47a90e530, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/a13ba8ca6dd54a569915e38d977706c3, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/d5b12b1efa8541b79e6ccfe88fa2bcb2, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/0b479bc616e14ddda34dbd7c544ef430, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/87d7ca87a0a941e8924b9df1eb907707, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/e3a0bc88f2734554bb553a1594e44c7c, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/e5b4664f4d57432eb4843aa729e8af6a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/b3b7d05686a74d62b2e7e8bd96c4a927, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/3608c8eff05b4550a28c3295bea58b56, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/0a2ead3b015b416580d4c9af8a6fca71, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/c552d48303514707a036ea1c2b0cb83d] to archive 2023-04-28 14:58:24,764 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 14:58:24,766 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/70d377b4e88a4cc3adbd1d7b18ba121a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/70d377b4e88a4cc3adbd1d7b18ba121a 2023-04-28 14:58:24,767 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/c9c6d4e6f7ba4c62ad47067533b6baf6 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/c9c6d4e6f7ba4c62ad47067533b6baf6 2023-04-28 14:58:24,767 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/f156cfd423824719b685280a380f831a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/f156cfd423824719b685280a380f831a 2023-04-28 14:58:24,768 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/7b5b284ced40412681ecd698d1098760 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/7b5b284ced40412681ecd698d1098760 2023-04-28 14:58:24,769 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/94d8d51e11534d59bf433357f9f830f1 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/94d8d51e11534d59bf433357f9f830f1 2023-04-28 14:58:24,770 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/64aa34bfcbea4485a9f771c9860e6cef to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/64aa34bfcbea4485a9f771c9860e6cef 2023-04-28 14:58:24,771 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/63bc9271a4ee4fe286cf9f7cefef0483 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/63bc9271a4ee4fe286cf9f7cefef0483 2023-04-28 14:58:24,772 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/ac2ecd90132941448718a1603cff8c42 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/ac2ecd90132941448718a1603cff8c42 2023-04-28 14:58:24,773 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/7a27d40375534ea5aba4d349aac90c32 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/7a27d40375534ea5aba4d349aac90c32 2023-04-28 14:58:24,775 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/d2627b4b297c481b8459a87a58bbfd99 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/d2627b4b297c481b8459a87a58bbfd99 2023-04-28 14:58:24,776 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/e2098ac32c3c4aabb78f456a864e2418 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/e2098ac32c3c4aabb78f456a864e2418 2023-04-28 14:58:24,776 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/15876e573f484482bc1ae855b967981b to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/15876e573f484482bc1ae855b967981b 2023-04-28 14:58:24,777 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/651de671b37e448db714e54c51162f6d to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/651de671b37e448db714e54c51162f6d 2023-04-28 14:58:24,778 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/5f3b4fd07a2a4f89bcc94c29c5858265 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/5f3b4fd07a2a4f89bcc94c29c5858265 2023-04-28 14:58:24,779 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/e5c0df22792a413b8dda6341feaafd5a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/e5c0df22792a413b8dda6341feaafd5a 2023-04-28 14:58:24,780 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/7f125540dcdd4e06ada696b564098d06 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/7f125540dcdd4e06ada696b564098d06 2023-04-28 14:58:24,781 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/a6fcc77810a64359ade5b8283483cdb1 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/a6fcc77810a64359ade5b8283483cdb1 2023-04-28 14:58:24,782 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/664c55fb574042ca9bf017a47a90e530 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/664c55fb574042ca9bf017a47a90e530 2023-04-28 14:58:24,784 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/a13ba8ca6dd54a569915e38d977706c3 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/a13ba8ca6dd54a569915e38d977706c3 2023-04-28 14:58:24,785 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/d5b12b1efa8541b79e6ccfe88fa2bcb2 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/d5b12b1efa8541b79e6ccfe88fa2bcb2 2023-04-28 14:58:24,786 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/0b479bc616e14ddda34dbd7c544ef430 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/0b479bc616e14ddda34dbd7c544ef430 2023-04-28 14:58:24,787 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/87d7ca87a0a941e8924b9df1eb907707 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/87d7ca87a0a941e8924b9df1eb907707 2023-04-28 14:58:24,788 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/e3a0bc88f2734554bb553a1594e44c7c to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/e3a0bc88f2734554bb553a1594e44c7c 2023-04-28 14:58:24,789 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/e5b4664f4d57432eb4843aa729e8af6a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/e5b4664f4d57432eb4843aa729e8af6a 2023-04-28 14:58:24,791 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/b3b7d05686a74d62b2e7e8bd96c4a927 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/b3b7d05686a74d62b2e7e8bd96c4a927 2023-04-28 14:58:24,792 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/3608c8eff05b4550a28c3295bea58b56 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/3608c8eff05b4550a28c3295bea58b56 2023-04-28 14:58:24,793 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/0a2ead3b015b416580d4c9af8a6fca71 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/0a2ead3b015b416580d4c9af8a6fca71 2023-04-28 14:58:24,794 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/c552d48303514707a036ea1c2b0cb83d to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/c552d48303514707a036ea1c2b0cb83d 2023-04-28 14:58:24,799 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/49cdd87bb5c047e384902806075cba30, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/08423da28b5c4b418e07807ad15ae8b7, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/fdd97e813e574ae79762ebc48678fb41, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/7f2dd54a6d87480383080f0131188611, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/3c8599f522ab4c278f8a691fe519fe30, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/f6110f1425ea4ac78a05b41dd3e2137a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/97ac3ad5d5fc4bd283fd4ef5a5c3200e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/96a1730e0f8949828aee839bde99ea15, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/6ace4e44c3ce479aaa2975c7d498f653, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/c1ef58e2a8b84d7d81680ba5e49448f6, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/2c4fb06a03e74bdeb5e9b84db8a509b7, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/491ac303e7b540ae85ad61232bfd36dd, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/4ad2bb2074a2413e9325584276381b30, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/24863ed37691480db93ce435d978d72b, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/0d38f92b82664883a5b0eaaaf66ccb0c, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/b0eb649b8d1e4fbca9932f731840a4c3, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/4b2245ebb6e2473796815bc1926ad140, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/68816356eb4144fdabd4dcb9e82a0b56, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/c8dae1cf5d444790a593d628800579ae, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/788b095fb93a4a2e91e2f228586503e1, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/7db1ba1de2dd473183e226ad49a8966d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/52ed6e1211424dd2a8d9fa4b1095e3c1, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/337426151a5c49f689052d8dbc9df36b, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/61faf19f89124b7aaafdf91ff288f2d4, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/91cab64113484cd7a629985979c5ed24, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/946082e587954e4eaee4b135ba134817, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/6d14313cb58b48f9a50d27699b92c251, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/76a89d2af30b44db928b9d761556566d] to archive 2023-04-28 14:58:24,800 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 14:58:24,802 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/49cdd87bb5c047e384902806075cba30 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/49cdd87bb5c047e384902806075cba30 2023-04-28 14:58:24,803 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/08423da28b5c4b418e07807ad15ae8b7 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/08423da28b5c4b418e07807ad15ae8b7 2023-04-28 14:58:24,804 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/fdd97e813e574ae79762ebc48678fb41 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/fdd97e813e574ae79762ebc48678fb41 2023-04-28 14:58:24,805 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/7f2dd54a6d87480383080f0131188611 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/7f2dd54a6d87480383080f0131188611 2023-04-28 14:58:24,805 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/3c8599f522ab4c278f8a691fe519fe30 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/3c8599f522ab4c278f8a691fe519fe30 2023-04-28 14:58:24,806 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/f6110f1425ea4ac78a05b41dd3e2137a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/f6110f1425ea4ac78a05b41dd3e2137a 2023-04-28 14:58:24,807 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/97ac3ad5d5fc4bd283fd4ef5a5c3200e to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/97ac3ad5d5fc4bd283fd4ef5a5c3200e 2023-04-28 14:58:24,808 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/96a1730e0f8949828aee839bde99ea15 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/96a1730e0f8949828aee839bde99ea15 2023-04-28 14:58:24,809 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/6ace4e44c3ce479aaa2975c7d498f653 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/6ace4e44c3ce479aaa2975c7d498f653 2023-04-28 14:58:24,809 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/c1ef58e2a8b84d7d81680ba5e49448f6 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/c1ef58e2a8b84d7d81680ba5e49448f6 2023-04-28 14:58:24,810 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/2c4fb06a03e74bdeb5e9b84db8a509b7 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/2c4fb06a03e74bdeb5e9b84db8a509b7 2023-04-28 14:58:24,811 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/491ac303e7b540ae85ad61232bfd36dd to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/491ac303e7b540ae85ad61232bfd36dd 2023-04-28 14:58:24,812 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/4ad2bb2074a2413e9325584276381b30 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/4ad2bb2074a2413e9325584276381b30 2023-04-28 14:58:24,813 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/24863ed37691480db93ce435d978d72b to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/24863ed37691480db93ce435d978d72b 2023-04-28 14:58:24,814 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/0d38f92b82664883a5b0eaaaf66ccb0c to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/0d38f92b82664883a5b0eaaaf66ccb0c 2023-04-28 14:58:24,815 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/b0eb649b8d1e4fbca9932f731840a4c3 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/b0eb649b8d1e4fbca9932f731840a4c3 2023-04-28 14:58:24,816 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/4b2245ebb6e2473796815bc1926ad140 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/4b2245ebb6e2473796815bc1926ad140 2023-04-28 14:58:24,817 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/68816356eb4144fdabd4dcb9e82a0b56 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/68816356eb4144fdabd4dcb9e82a0b56 2023-04-28 14:58:24,819 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/c8dae1cf5d444790a593d628800579ae to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/c8dae1cf5d444790a593d628800579ae 2023-04-28 14:58:24,820 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/788b095fb93a4a2e91e2f228586503e1 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/788b095fb93a4a2e91e2f228586503e1 2023-04-28 14:58:24,820 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/7db1ba1de2dd473183e226ad49a8966d to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/7db1ba1de2dd473183e226ad49a8966d 2023-04-28 14:58:24,822 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/52ed6e1211424dd2a8d9fa4b1095e3c1 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/52ed6e1211424dd2a8d9fa4b1095e3c1 2023-04-28 14:58:24,824 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/337426151a5c49f689052d8dbc9df36b to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/337426151a5c49f689052d8dbc9df36b 2023-04-28 14:58:24,825 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/61faf19f89124b7aaafdf91ff288f2d4 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/61faf19f89124b7aaafdf91ff288f2d4 2023-04-28 14:58:24,826 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/91cab64113484cd7a629985979c5ed24 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/91cab64113484cd7a629985979c5ed24 2023-04-28 14:58:24,828 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/946082e587954e4eaee4b135ba134817 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/946082e587954e4eaee4b135ba134817 2023-04-28 14:58:24,829 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/6d14313cb58b48f9a50d27699b92c251 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/6d14313cb58b48f9a50d27699b92c251 2023-04-28 14:58:24,830 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/76a89d2af30b44db928b9d761556566d to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/76a89d2af30b44db928b9d761556566d 2023-04-28 14:58:24,833 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/bb7a797e4df14b8fb8a452aeb40ef9dc, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/47c6c7e4f12941a89435f77142b14f7d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/067cced26b5343c59bca5e0fdbb1366d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/76a6167544fa43fcb34e87cd498a36c3, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/2522d83b91f8482587078da78437f824, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/fcb2840355cd4db49c8d5ea1461284fa, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/164cdd2be30e424692f1d6646d33c914, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/c89dcbf29ed242cc912d4a3763ea18f8, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/7668bf346c1a436c908bae1e232fa3b0, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/fd4c090f866242f0a419540aa3de8e56, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/a22526c414c5406886bae20ba16b4ade, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/afc207f45af141c7b714ac7f60d1e8e1, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/5b6c5c9671554807b0b64c75604e9e78, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/3b4c92935cfa43dda9256e561158472a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/4bc46f16287842ac8ed387e9fce2fec1, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/e54fa757d8064e66830ce7350f7b371e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/655edcf8431b455a9b840681bff3388d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/8c2e25a06a1e402d8560d77d242e63ad, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/c6cd7fea1aa24292936a77093ec3354e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/027700f694d74253ba9f3a3994e1a491, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/6ee09e2ef88a4cf5963ebf6c2fca9edc, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/292aa52d0a5f4e89b8243071aee3d2ca, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/d3665f247c9a4e53a10b851c670946e2, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/06582355a1cf48688ecd82b6dd274494, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/96f992860b9d4524bf3852f73349996a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/276d88797e8141dc8afda128852737d5, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/19c868be962d48e7a6c0412441c94c2f, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/fba04e031cb0455fbbf4dadb3e849d70] to archive 2023-04-28 14:58:24,834 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 14:58:24,835 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/bb7a797e4df14b8fb8a452aeb40ef9dc to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/bb7a797e4df14b8fb8a452aeb40ef9dc 2023-04-28 14:58:24,836 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/47c6c7e4f12941a89435f77142b14f7d to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/47c6c7e4f12941a89435f77142b14f7d 2023-04-28 14:58:24,837 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/067cced26b5343c59bca5e0fdbb1366d to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/067cced26b5343c59bca5e0fdbb1366d 2023-04-28 14:58:24,838 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/76a6167544fa43fcb34e87cd498a36c3 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/76a6167544fa43fcb34e87cd498a36c3 2023-04-28 14:58:24,839 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/2522d83b91f8482587078da78437f824 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/2522d83b91f8482587078da78437f824 2023-04-28 14:58:24,839 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/fcb2840355cd4db49c8d5ea1461284fa to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/fcb2840355cd4db49c8d5ea1461284fa 2023-04-28 14:58:24,840 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/164cdd2be30e424692f1d6646d33c914 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/164cdd2be30e424692f1d6646d33c914 2023-04-28 14:58:24,841 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/c89dcbf29ed242cc912d4a3763ea18f8 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/c89dcbf29ed242cc912d4a3763ea18f8 2023-04-28 14:58:24,841 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/7668bf346c1a436c908bae1e232fa3b0 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/7668bf346c1a436c908bae1e232fa3b0 2023-04-28 14:58:24,842 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/fd4c090f866242f0a419540aa3de8e56 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/fd4c090f866242f0a419540aa3de8e56 2023-04-28 14:58:24,843 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/a22526c414c5406886bae20ba16b4ade to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/a22526c414c5406886bae20ba16b4ade 2023-04-28 14:58:24,844 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/afc207f45af141c7b714ac7f60d1e8e1 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/afc207f45af141c7b714ac7f60d1e8e1 2023-04-28 14:58:24,844 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/5b6c5c9671554807b0b64c75604e9e78 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/5b6c5c9671554807b0b64c75604e9e78 2023-04-28 14:58:24,845 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/3b4c92935cfa43dda9256e561158472a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/3b4c92935cfa43dda9256e561158472a 2023-04-28 14:58:24,846 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/4bc46f16287842ac8ed387e9fce2fec1 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/4bc46f16287842ac8ed387e9fce2fec1 2023-04-28 14:58:24,846 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/e54fa757d8064e66830ce7350f7b371e to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/e54fa757d8064e66830ce7350f7b371e 2023-04-28 14:58:24,847 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/655edcf8431b455a9b840681bff3388d to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/655edcf8431b455a9b840681bff3388d 2023-04-28 14:58:24,848 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/8c2e25a06a1e402d8560d77d242e63ad to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/8c2e25a06a1e402d8560d77d242e63ad 2023-04-28 14:58:24,848 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/c6cd7fea1aa24292936a77093ec3354e to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/c6cd7fea1aa24292936a77093ec3354e 2023-04-28 14:58:24,849 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/027700f694d74253ba9f3a3994e1a491 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/027700f694d74253ba9f3a3994e1a491 2023-04-28 14:58:24,849 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/6ee09e2ef88a4cf5963ebf6c2fca9edc to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/6ee09e2ef88a4cf5963ebf6c2fca9edc 2023-04-28 14:58:24,850 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/292aa52d0a5f4e89b8243071aee3d2ca to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/292aa52d0a5f4e89b8243071aee3d2ca 2023-04-28 14:58:24,851 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/d3665f247c9a4e53a10b851c670946e2 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/d3665f247c9a4e53a10b851c670946e2 2023-04-28 14:58:24,851 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/06582355a1cf48688ecd82b6dd274494 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/06582355a1cf48688ecd82b6dd274494 2023-04-28 14:58:24,852 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/96f992860b9d4524bf3852f73349996a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/96f992860b9d4524bf3852f73349996a 2023-04-28 14:58:24,852 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/276d88797e8141dc8afda128852737d5 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/276d88797e8141dc8afda128852737d5 2023-04-28 14:58:24,853 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/19c868be962d48e7a6c0412441c94c2f to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/19c868be962d48e7a6c0412441c94c2f 2023-04-28 14:58:24,854 DEBUG [StoreCloser-TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/fba04e031cb0455fbbf4dadb3e849d70 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/fba04e031cb0455fbbf4dadb3e849d70 2023-04-28 14:58:24,858 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/recovered.edits/621.seqid, newMaxSeqId=621, maxSeqId=1 2023-04-28 14:58:24,859 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8. 2023-04-28 14:58:24,859 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1558): Region close journal for f40347916069de574ae857f0751743f8: 2023-04-28 14:58:24,860 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(147): Closed f40347916069de574ae857f0751743f8 2023-04-28 14:58:24,860 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=f40347916069de574ae857f0751743f8, regionState=CLOSED 2023-04-28 14:58:24,861 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693904860"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682693904860"}]},"ts":"1682693904860"} 2023-04-28 14:58:24,862 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-28 14:58:24,862 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure f40347916069de574ae857f0751743f8, server=jenkins-hbase17.apache.org,37181,1682693812482 in 2.5170 sec 2023-04-28 14:58:24,864 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-28 14:58:24,864 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=f40347916069de574ae857f0751743f8, UNASSIGN in 2.5220 sec 2023-04-28 14:58:24,864 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682693904864"}]},"ts":"1682693904864"} 2023-04-28 14:58:24,865 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-28 14:58:24,865 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-28 14:58:24,867 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.5380 sec 2023-04-28 14:58:26,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-28 14:58:26,447 INFO [Listener at localhost.localdomain/40193] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-28 14:58:26,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.HMaster$5(2228): Client=jenkins//136.243.18.41 delete TestAcidGuarantees 2023-04-28 14:58:26,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:58:26,454 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:58:26,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-28 14:58:26,454 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:58:26,458 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8 2023-04-28 14:58:26,460 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A, FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B, FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C, FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/recovered.edits] 2023-04-28 14:58:26,462 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/439a180c6db24b3db50091acca8fa838 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/439a180c6db24b3db50091acca8fa838 2023-04-28 14:58:26,463 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/488fb78be30d49a88e374233c562547b to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/488fb78be30d49a88e374233c562547b 2023-04-28 14:58:26,464 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/5cc6ef6e25b34841acc22852092f57ff to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/5cc6ef6e25b34841acc22852092f57ff 2023-04-28 14:58:26,465 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/67ba696503a24251a4e79d7c345d6320 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/67ba696503a24251a4e79d7c345d6320 2023-04-28 14:58:26,466 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/7f0dd84c7caf4d008daae4eeae4ba4de to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/7f0dd84c7caf4d008daae4eeae4ba4de 2023-04-28 14:58:26,467 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/857bb5128b7645c59f2b16ac34a79b9c to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/A/857bb5128b7645c59f2b16ac34a79b9c 2023-04-28 14:58:26,469 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/26ffc1fb8107424db01556d0da074841 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/26ffc1fb8107424db01556d0da074841 2023-04-28 14:58:26,470 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/44775bd721ed4269b21562b504523567 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/44775bd721ed4269b21562b504523567 2023-04-28 14:58:26,471 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/5de777c730b545d0af8b98e937c16d38 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/5de777c730b545d0af8b98e937c16d38 2023-04-28 14:58:26,471 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/7a803a4346d349739a699732238fbe88 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/7a803a4346d349739a699732238fbe88 2023-04-28 14:58:26,472 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/aae4423866f54539a73941137f501a6a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/aae4423866f54539a73941137f501a6a 2023-04-28 14:58:26,473 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/e347f0d08a0a411fb6f18a957a36fb89 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/B/e347f0d08a0a411fb6f18a957a36fb89 2023-04-28 14:58:26,475 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/2661ac2e3e62452db8c0d30643b53837 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/2661ac2e3e62452db8c0d30643b53837 2023-04-28 14:58:26,476 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/50f2348452114b26a98ff5f3b5004ab6 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/50f2348452114b26a98ff5f3b5004ab6 2023-04-28 14:58:26,476 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/612acf70959d47aebf41bae37f083329 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/612acf70959d47aebf41bae37f083329 2023-04-28 14:58:26,477 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/747c0bad303f47f5800307f7a017e1a7 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/747c0bad303f47f5800307f7a017e1a7 2023-04-28 14:58:26,478 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/876da018e8ff4b68b2164bea8e3869e0 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/876da018e8ff4b68b2164bea8e3869e0 2023-04-28 14:58:26,479 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/b7774ace84224ae7b4e3d123bb0d3200 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/C/b7774ace84224ae7b4e3d123bb0d3200 2023-04-28 14:58:26,480 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/recovered.edits/621.seqid to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8/recovered.edits/621.seqid 2023-04-28 14:58:26,481 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/f40347916069de574ae857f0751743f8 2023-04-28 14:58:26,481 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 14:58:26,482 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:58:26,483 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-28 14:58:26,485 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-28 14:58:26,486 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:58:26,486 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-28 14:58:26,486 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682693906486"}]},"ts":"9223372036854775807"} 2023-04-28 14:58:26,487 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-28 14:58:26,487 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => f40347916069de574ae857f0751743f8, NAME => 'TestAcidGuarantees,,1682693879886.f40347916069de574ae857f0751743f8.', STARTKEY => '', ENDKEY => ''}] 2023-04-28 14:58:26,487 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-28 14:58:26,487 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682693906487"}]},"ts":"9223372036854775807"} 2023-04-28 14:58:26,488 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-28 14:58:26,490 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:58:26,491 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 41 msec 2023-04-28 14:58:26,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-28 14:58:26,555 INFO [Listener at localhost.localdomain/40193] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-28 14:58:26,566 INFO [Listener at localhost.localdomain/40193] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=227 (was 224) - Thread LEAK? -, OpenFileDescriptor=566 (was 501) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=364 (was 332) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=4925 (was 5085) 2023-04-28 14:58:26,576 INFO [Listener at localhost.localdomain/40193] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227, OpenFileDescriptor=566, MaxFileDescriptor=60000, SystemLoadAverage=364, ProcessCount=168, AvailableMemoryMB=4925 2023-04-28 14:58:26,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-28 14:58:26,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.HMaster$4(2112): Client=jenkins//136.243.18.41 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-28 14:58:26,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-28 14:58:26,581 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-28 14:58:26,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(697): Client=jenkins//136.243.18.41 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-28 14:58:26,581 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-28 14:58:26,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-28 14:58:26,582 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:26,583 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a empty. 2023-04-28 14:58:26,583 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:26,584 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 14:58:26,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-28 14:58:26,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-28 14:58:26,995 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-28 14:58:26,998 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 1b6356a2d73db55249b3f0cb6bbd108a, NAME => 'TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp 2023-04-28 14:58:27,007 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 14:58:27,008 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 1b6356a2d73db55249b3f0cb6bbd108a, disabling compactions & flushes 2023-04-28 14:58:27,008 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:27,008 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:27,008 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. after waiting 0 ms 2023-04-28 14:58:27,008 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:27,008 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:27,008 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:27,010 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-28 14:58:27,010 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693907010"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682693907010"}]},"ts":"1682693907010"} 2023-04-28 14:58:27,011 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-28 14:58:27,012 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-28 14:58:27,012 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682693907012"}]},"ts":"1682693907012"} 2023-04-28 14:58:27,013 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-28 14:58:27,015 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1b6356a2d73db55249b3f0cb6bbd108a, ASSIGN}] 2023-04-28 14:58:27,016 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1b6356a2d73db55249b3f0cb6bbd108a, ASSIGN 2023-04-28 14:58:27,016 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=1b6356a2d73db55249b3f0cb6bbd108a, ASSIGN; state=OFFLINE, location=jenkins-hbase17.apache.org,37181,1682693812482; forceNewPlan=false, retain=false 2023-04-28 14:58:27,168 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=1b6356a2d73db55249b3f0cb6bbd108a, regionState=OPENING, regionLocation=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:27,168 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693907168"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682693907168"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682693907168"}]},"ts":"1682693907168"} 2023-04-28 14:58:27,171 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482}] 2023-04-28 14:58:27,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-28 14:58:27,328 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:27,328 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1b6356a2d73db55249b3f0cb6bbd108a, NAME => 'TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a.', STARTKEY => '', ENDKEY => ''} 2023-04-28 14:58:27,329 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:27,329 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 14:58:27,329 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7894): checking encryption for 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:27,329 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7897): checking classloading for 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:27,330 INFO [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:27,333 DEBUG [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A 2023-04-28 14:58:27,333 DEBUG [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A 2023-04-28 14:58:27,333 INFO [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 14:58:27,333 INFO [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1b6356a2d73db55249b3f0cb6bbd108a columnFamilyName A 2023-04-28 14:58:27,334 INFO [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] regionserver.HStore(310): Store=1b6356a2d73db55249b3f0cb6bbd108a/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:58:27,334 INFO [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:27,336 DEBUG [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B 2023-04-28 14:58:27,336 DEBUG [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B 2023-04-28 14:58:27,337 INFO [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 14:58:27,337 INFO [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1b6356a2d73db55249b3f0cb6bbd108a columnFamilyName B 2023-04-28 14:58:27,338 INFO [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] regionserver.HStore(310): Store=1b6356a2d73db55249b3f0cb6bbd108a/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:58:27,338 INFO [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:27,340 DEBUG [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C 2023-04-28 14:58:27,340 DEBUG [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C 2023-04-28 14:58:27,340 INFO [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 14:58:27,341 INFO [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1b6356a2d73db55249b3f0cb6bbd108a columnFamilyName C 2023-04-28 14:58:27,341 INFO [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] regionserver.HStore(310): Store=1b6356a2d73db55249b3f0cb6bbd108a/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:58:27,341 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:27,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:27,343 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:27,345 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 14:58:27,347 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1055): writing seq id for 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:27,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-28 14:58:27,350 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1072): Opened 1b6356a2d73db55249b3f0cb6bbd108a; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71520873, jitterRate=0.0657440572977066}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 14:58:27,350 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(965): Region open journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:27,350 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a., pid=37, masterSystemTime=1682693907324 2023-04-28 14:58:27,352 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:27,353 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:27,353 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=1b6356a2d73db55249b3f0cb6bbd108a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:27,353 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693907353"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1682693907353"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682693907353"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682693907353"}]},"ts":"1682693907353"} 2023-04-28 14:58:27,357 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-28 14:58:27,357 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 in 184 msec 2023-04-28 14:58:27,359 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-28 14:58:27,359 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1b6356a2d73db55249b3f0cb6bbd108a, ASSIGN in 342 msec 2023-04-28 14:58:27,359 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-28 14:58:27,360 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682693907360"}]},"ts":"1682693907360"} 2023-04-28 14:58:27,361 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-28 14:58:27,363 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-28 14:58:27,365 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 784 msec 2023-04-28 14:58:27,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-28 14:58:27,691 INFO [Listener at localhost.localdomain/40193] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-28 14:58:27,693 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x7d243e46 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:58:27,702 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29738198, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:58:27,703 DEBUG [Listener at localhost.localdomain/40193] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:58:27,705 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:37410, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:58:27,706 DEBUG [Listener at localhost.localdomain/40193] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-28 14:58:27,707 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:37982, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-28 14:58:27,708 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-28 14:58:27,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.HMaster$12(2454): Client=jenkins//136.243.18.41 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-28 14:58:27,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-28 14:58:28,119 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-28 14:58:28,121 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-28 14:58:28,121 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-28 14:58:28,124 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-28 14:58:28,129 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1b6356a2d73db55249b3f0cb6bbd108a, REOPEN/MOVE}] 2023-04-28 14:58:28,130 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1b6356a2d73db55249b3f0cb6bbd108a, REOPEN/MOVE 2023-04-28 14:58:28,130 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=1b6356a2d73db55249b3f0cb6bbd108a, regionState=CLOSING, regionLocation=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,130 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693908130"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682693908130"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682693908130"}]},"ts":"1682693908130"} 2023-04-28 14:58:28,131 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482}] 2023-04-28 14:58:28,283 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(109): Close 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:28,284 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1604): Closing 1b6356a2d73db55249b3f0cb6bbd108a, disabling compactions & flushes 2023-04-28 14:58:28,284 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:28,284 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:28,284 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. after waiting 0 ms 2023-04-28 14:58:28,284 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:28,290 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-28 14:58:28,291 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:28,291 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1558): Region close journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:28,291 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 1b6356a2d73db55249b3f0cb6bbd108a to self. 2023-04-28 14:58:28,294 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(147): Closed 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:28,294 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=1b6356a2d73db55249b3f0cb6bbd108a, regionState=CLOSED 2023-04-28 14:58:28,294 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693908294"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682693908294"}]},"ts":"1682693908294"} 2023-04-28 14:58:28,299 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-28 14:58:28,299 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 in 165 msec 2023-04-28 14:58:28,300 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=1b6356a2d73db55249b3f0cb6bbd108a, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase17.apache.org,37181,1682693812482; forceNewPlan=false, retain=true 2023-04-28 14:58:28,451 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=1b6356a2d73db55249b3f0cb6bbd108a, regionState=OPENING, regionLocation=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,451 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693908451"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682693908451"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682693908451"}]},"ts":"1682693908451"} 2023-04-28 14:58:28,454 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482}] 2023-04-28 14:58:28,615 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:28,615 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1b6356a2d73db55249b3f0cb6bbd108a, NAME => 'TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a.', STARTKEY => '', ENDKEY => ''} 2023-04-28 14:58:28,616 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:28,616 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 14:58:28,616 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7894): checking encryption for 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:28,616 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7897): checking classloading for 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:28,618 INFO [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:28,619 DEBUG [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A 2023-04-28 14:58:28,619 DEBUG [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A 2023-04-28 14:58:28,619 INFO [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 14:58:28,619 INFO [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1b6356a2d73db55249b3f0cb6bbd108a columnFamilyName A 2023-04-28 14:58:28,621 INFO [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] regionserver.HStore(310): Store=1b6356a2d73db55249b3f0cb6bbd108a/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:58:28,621 INFO [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:28,621 DEBUG [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B 2023-04-28 14:58:28,621 DEBUG [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B 2023-04-28 14:58:28,622 INFO [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 14:58:28,622 INFO [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1b6356a2d73db55249b3f0cb6bbd108a columnFamilyName B 2023-04-28 14:58:28,622 INFO [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] regionserver.HStore(310): Store=1b6356a2d73db55249b3f0cb6bbd108a/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:58:28,622 INFO [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:28,623 DEBUG [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C 2023-04-28 14:58:28,623 DEBUG [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C 2023-04-28 14:58:28,623 INFO [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 14:58:28,623 INFO [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1b6356a2d73db55249b3f0cb6bbd108a columnFamilyName C 2023-04-28 14:58:28,624 INFO [StoreOpener-1b6356a2d73db55249b3f0cb6bbd108a-1] regionserver.HStore(310): Store=1b6356a2d73db55249b3f0cb6bbd108a/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:58:28,624 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:28,625 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:28,625 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:28,627 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 14:58:28,628 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1055): writing seq id for 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:28,628 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1072): Opened 1b6356a2d73db55249b3f0cb6bbd108a; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67793302, jitterRate=0.010198920965194702}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 14:58:28,629 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(965): Region open journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:28,629 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a., pid=42, masterSystemTime=1682693908607 2023-04-28 14:58:28,630 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:28,630 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:28,630 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=1b6356a2d73db55249b3f0cb6bbd108a, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,631 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693908630"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1682693908630"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682693908630"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682693908630"}]},"ts":"1682693908630"} 2023-04-28 14:58:28,632 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-28 14:58:28,632 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 in 177 msec 2023-04-28 14:58:28,634 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-28 14:58:28,634 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1b6356a2d73db55249b3f0cb6bbd108a, REOPEN/MOVE in 503 msec 2023-04-28 14:58:28,635 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-28 14:58:28,635 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 510 msec 2023-04-28 14:58:28,636 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 926 msec 2023-04-28 14:58:28,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-28 14:58:28,638 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x74a355de to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:58:28,641 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28a98302, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:58:28,642 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x57b6628e to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:58:28,648 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@39d5c003, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:58:28,649 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x5da5030e to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:58:28,653 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@39e3a826, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:58:28,653 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x74d6bf18 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:58:28,657 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@19e7fff, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:58:28,657 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x060c5cba to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:58:28,661 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@47e30d1c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:58:28,661 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x36a9ece3 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:58:28,665 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69fd9cc7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:58:28,665 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x2ded66ad to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:58:28,668 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54da4907, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:58:28,669 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x14d061d2 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:58:28,674 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52cea5eb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:58:28,674 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x228ff1a4 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:58:28,677 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3546dad6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:58:28,678 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x143a1290 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:58:28,681 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4965c362, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:58:28,686 DEBUG [hconnection-0x2e7eff9c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:58:28,686 DEBUG [hconnection-0x5e7c2f03-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:58:28,687 DEBUG [hconnection-0x33699f0e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:58:28,687 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:37412, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:58:28,687 DEBUG [hconnection-0xb3a9e47-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:58:28,688 DEBUG [hconnection-0x3ecc1040-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:58:28,691 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:37426, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:58:28,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:28,692 DEBUG [hconnection-0x11f4a228-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:58:28,692 DEBUG [hconnection-0x38aa2d1b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:58:28,694 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:37436, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:58:28,694 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:37424, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:58:28,695 DEBUG [hconnection-0x6a9f2b6b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:58:28,697 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:37454, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:58:28,696 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:37468, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:58:28,696 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:37450, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:58:28,696 DEBUG [hconnection-0x1a9829d4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:58:28,698 DEBUG [hconnection-0x921b576-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:58:28,703 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:37480, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:58:28,703 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:37478, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:58:28,703 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:37496, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:58:28,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:28,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:28,708 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:28,708 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:28,708 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:28,708 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:28,709 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:28,709 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:28,713 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,713 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:28,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:28,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:28,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:28,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:28,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:28,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:28,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:28,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:28,715 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:28,715 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1b6356a2d73db55249b3f0cb6bbd108a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 14:58:28,715 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:28,715 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:28,715 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:28,715 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:28,716 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:28,716 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:28,716 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:28,716 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:28,716 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:28,716 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:28,716 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:28,716 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:28,716 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,716 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:28,716 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:28,716 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,716 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:28,716 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=A 2023-04-28 14:58:28,716 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:28,716 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=B 2023-04-28 14:58:28,716 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:28,717 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=C 2023-04-28 14:58:28,717 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:28,717 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:28,717 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:28,717 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:28,718 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:28,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:28,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:28,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:28,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:28,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:28,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e007bf4[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:28,724 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:28,724 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:28,724 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:28,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:28,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:28,725 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:28,725 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,725 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:28,725 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:28,725 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:28,726 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:28,726 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,726 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:28,727 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:28,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:28,728 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:28,729 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:28,729 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:28,729 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:28,729 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:28,729 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:28,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:28,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:28,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:28,730 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:28,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:28,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37450 deadline: 1682693968729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,730 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:28,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37454 deadline: 1682693968729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,731 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:28,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37496 deadline: 1682693968730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,731 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:28,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:28,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:28,731 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:28,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:28,731 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682693908723, max timestamp=1682693908729Num uniques 859; 2023-04-28 14:58:28,731 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682693908723, max timestamp=1682693908729Num uniques 859; 2023-04-28 14:58:28,731 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-28 14:58:28,731 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682693908723, max timestamp=1682693908729Num uniques 859; 2023-04-28 14:58:28,731 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-28 14:58:28,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:28,731 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-28 14:58:28,731 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:28,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:28,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:28,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:28,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:28,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:28,736 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:28,736 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,736 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:28,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:28,736 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:28,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:28,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:28,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:28,736 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:28,737 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:28,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:28,737 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,737 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:28,737 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:28,737 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:28,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:28,737 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:28,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:28,737 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:28,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:28,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37424 deadline: 1682693968754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,759 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:28,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37478 deadline: 1682693968759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:28,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37454 deadline: 1682693968831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:28,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37496 deadline: 1682693968831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:28,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37450 deadline: 1682693968832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,837 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:28,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:28,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:28,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:28,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:28,842 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:28,842 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:28,842 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:28,842 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:28,842 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:28,842 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:28,843 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,843 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:28,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:28,843 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,843 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:28,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:28,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:28,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:28,844 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:28,844 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:28,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:28,844 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:28,844 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:28,844 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:28,844 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:28,844 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:28,844 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:28,844 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:28,845 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:28,845 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:28,845 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:28,845 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:28,845 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,845 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:28,845 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:28,846 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,846 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:28,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:28,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:28,847 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:28,847 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:28,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:28,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:28,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:28,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:28,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:28,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a0b6071[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:28,848 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:28,848 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:28,849 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:28,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:28,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:28,849 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:28,849 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:28,849 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,849 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:28,849 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:28,850 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:28,850 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,850 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:28,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:28,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:28,850 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:28,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:28,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:28,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:28,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:28,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:28,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:28,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:28,852 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:28,852 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:28,852 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:28,852 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:28,852 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:28,853 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:28,853 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:28,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:28,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:28,853 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:28,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:28,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:28,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:28,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:28,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:28,853 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:28,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:28,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:28,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:28,856 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:28,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:28,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:28,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,857 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,857 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:28,857 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:28,857 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,857 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:28,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:28,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:28,857 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:28,858 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:28,857 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:28,858 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:28,858 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:28,858 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:28,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:28,858 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:28,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:28,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:28,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:28,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:28,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:28,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37424 deadline: 1682693968860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:28,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37478 deadline: 1682693968860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,958 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:28,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:28,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:28,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:28,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:28,961 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:28,961 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:28,961 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:28,961 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:28,961 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:28,961 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:28,962 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:28,962 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:28,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:28,962 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,963 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:28,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:28,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:28,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:28,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:28,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:28,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:28,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:28,963 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:28,964 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:28,964 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:28,966 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:28,966 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:28,966 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:28,966 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:28,966 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:28,966 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:28,966 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:28,966 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:28,966 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,966 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:28,966 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:28,967 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:28,967 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,967 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:28,967 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:28,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:28,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:28,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:28,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:28,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:28,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:28,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:28,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@354a93bf[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:28,969 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:28,969 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:28,970 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:28,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:28,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:28,970 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:28,970 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:28,970 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,970 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:28,970 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:28,971 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,971 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:28,971 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:28,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:28,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:28,971 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:28,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:28,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:28,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:28,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:28,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:28,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:28,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:28,972 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:28,972 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:28,972 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:28,972 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:28,972 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:28,973 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:28,973 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:28,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:28,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:28,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:28,973 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:28,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:28,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:28,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:28,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:28,974 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:28,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:28,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:28,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:28,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:28,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,976 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:28,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:28,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:28,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:28,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:28,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:28,977 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:28,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:28,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:28,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:28,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:28,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:28,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:28,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:28,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:28,977 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:28,977 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:28,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:28,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:29,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:29,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37496 deadline: 1682693969036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:29,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37454 deadline: 1682693969037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:29,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37450 deadline: 1682693969037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:29,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37424 deadline: 1682693969061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:29,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37478 deadline: 1682693969062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,078 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:29,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:29,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:29,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:29,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:29,080 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:29,080 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:29,080 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:29,080 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:29,080 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,080 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:29,081 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,081 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:29,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:29,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:29,081 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,081 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:29,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:29,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:29,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:29,082 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:29,082 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:29,082 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:29,082 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:29,082 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:29,082 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:29,082 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:29,082 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:29,083 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:29,083 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:29,083 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:29,083 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:29,083 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:29,083 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:29,083 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,083 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:29,083 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:29,083 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,084 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:29,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:29,084 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,084 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:29,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:29,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:29,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:29,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:29,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:29,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@416714b4[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:29,085 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:29,085 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,085 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,086 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:29,086 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:29,086 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:29,086 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,086 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:29,086 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:29,086 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,086 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:29,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,086 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:29,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:29,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:29,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:29,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:29,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:29,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:29,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:29,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:29,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:29,090 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:29,090 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:29,090 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:29,090 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:29,090 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:29,094 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,094 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:29,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:29,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,094 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:29,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:29,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:29,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:29,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:29,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:29,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:29,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:29,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:29,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,096 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,097 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,098 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,098 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:29,098 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:29,098 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:29,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:29,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:29,098 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:29,098 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,098 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:29,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:29,099 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:29,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:29,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,099 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:29,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:29,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:29,140 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428e612a12b266f4d88bd9fd38086e9a472 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428e612a12b266f4d88bd9fd38086e9a472 2023-04-28 14:58:29,144 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/8cc291db9d1e4b5aaf853dc2ebbb8272 2023-04-28 14:58:29,199 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:29,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:29,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:29,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:29,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:29,201 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:29,202 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:29,202 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:29,202 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:29,202 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,202 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:29,202 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,202 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:29,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:29,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:29,203 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,203 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:29,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:29,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:29,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:29,203 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:29,203 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:29,203 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:29,204 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:29,204 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:29,204 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:29,204 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:29,204 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:29,204 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:29,204 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:29,204 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:29,204 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:29,204 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:29,204 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:29,204 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,204 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:29,204 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:29,205 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,205 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:29,205 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,205 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:29,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:29,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c0508b7[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:29,206 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:29,206 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,207 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,207 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:29,207 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:29,207 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:29,207 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,207 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:29,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:29,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:29,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:29,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:29,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:29,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:29,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:29,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:29,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:29,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:29,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:29,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:29,210 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:29,210 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:29,210 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:29,210 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:29,210 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:29,211 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,211 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:29,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,211 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:29,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:29,211 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:29,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:29,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,212 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:29,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:29,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,213 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:29,214 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,214 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:29,214 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:29,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:29,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:29,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:29,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:29,214 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:29,214 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:29,214 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:29,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:29,214 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:29,214 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,214 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:29,214 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,214 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,314 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:29,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:29,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:29,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:29,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:29,317 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:29,317 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:29,317 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:29,317 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:29,317 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,317 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:29,318 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:29,318 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:29,318 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,318 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:29,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:29,319 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:29,319 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:29,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:29,319 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:29,319 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:29,319 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:29,319 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:29,319 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:29,319 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:29,319 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:29,319 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:29,319 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:29,320 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:29,320 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:29,320 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,320 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:29,320 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:29,321 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,321 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:29,321 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,321 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:29,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:29,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d3ce7ae[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:29,323 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:29,323 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,323 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:29,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:29,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:29,323 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,323 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:29,324 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,324 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:29,324 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:29,324 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:29,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:29,326 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:29,326 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:29,326 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:29,326 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:29,326 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:29,327 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,327 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:29,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:29,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,327 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:29,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:29,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:29,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:29,327 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:29,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:29,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:29,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,328 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:29,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:29,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,330 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,330 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:29,330 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:29,330 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:29,330 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:29,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:29,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:29,330 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,330 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:29,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:29,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:29,331 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:29,331 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:29,330 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:29,330 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:29,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:29,331 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:29,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37496 deadline: 1682693969339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:29,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37450 deadline: 1682693969339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:29,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37454 deadline: 1682693969340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:29,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37424 deadline: 1682693969365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,369 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:29,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37478 deadline: 1682693969366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,431 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:29,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:29,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:29,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:29,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:29,433 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:29,433 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:29,433 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:29,433 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:29,433 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,434 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:29,434 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,434 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:29,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:29,434 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,434 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:29,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:29,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:29,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:29,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:29,435 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:29,435 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:29,435 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:29,435 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:29,435 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:29,435 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:29,436 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:29,436 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:29,436 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:29,436 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:29,436 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:29,436 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,436 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:29,436 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:29,436 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,437 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:29,437 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,437 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:29,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:29,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c8a52dd[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:29,438 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:29,438 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,438 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,439 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:29,439 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,439 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:29,439 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:29,439 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:29,439 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:29,439 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:29,439 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,439 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:29,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:29,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:29,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:29,441 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:29,441 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:29,441 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:29,441 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:29,441 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:29,442 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,442 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,442 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:29,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:29,442 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:29,443 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:29,445 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,445 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:29,445 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:29,445 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,445 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,445 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:29,445 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:29,445 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,445 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:29,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:29,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:29,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:29,445 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:29,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:29,445 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:29,445 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:29,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:29,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:29,545 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:29,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:29,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:29,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:29,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:29,548 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:29,548 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:29,548 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:29,548 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:29,549 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,549 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:29,549 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:29,549 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:29,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:29,550 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,550 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:29,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:29,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:29,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:29,550 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:29,550 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:29,551 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:29,551 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:29,551 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:29,551 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:29,551 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:29,551 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:29,551 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:29,551 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:29,551 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:29,551 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:29,551 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:29,551 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:29,551 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,551 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:29,551 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:29,552 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,552 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,552 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/f47036c560f64891946b5871cfadbc26 2023-04-28 14:58:29,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:29,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:29,553 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,553 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:29,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:29,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:29,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:29,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:29,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:29,554 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:29,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55b1214c[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:29,554 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,555 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,555 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:29,555 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:29,555 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:29,555 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,555 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:29,556 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:29,556 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:29,556 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:29,557 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:29,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:29,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:29,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:29,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:29,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:29,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:29,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:29,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:29,560 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:29,560 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:29,560 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:29,560 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:29,560 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:29,561 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/b70b8d11e45945879fa01ae59008878c 2023-04-28 14:58:29,562 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,562 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:29,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:29,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,562 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:29,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:29,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:29,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:29,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,563 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:29,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:29,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:29,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,564 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:29,564 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/8cc291db9d1e4b5aaf853dc2ebbb8272 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/8cc291db9d1e4b5aaf853dc2ebbb8272 2023-04-28 14:58:29,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:29,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,566 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,566 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:29,566 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:29,566 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,566 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,566 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:29,566 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,566 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:29,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:29,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:29,566 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:29,566 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:29,568 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/8cc291db9d1e4b5aaf853dc2ebbb8272, entries=200, sequenceid=15, filesize=31.9 K 2023-04-28 14:58:29,568 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/f47036c560f64891946b5871cfadbc26 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/f47036c560f64891946b5871cfadbc26 2023-04-28 14:58:29,569 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:29,569 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:29,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:29,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:29,572 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/f47036c560f64891946b5871cfadbc26, entries=150, sequenceid=15, filesize=11.6 K 2023-04-28 14:58:29,572 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/b70b8d11e45945879fa01ae59008878c as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/b70b8d11e45945879fa01ae59008878c 2023-04-28 14:58:29,576 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/b70b8d11e45945879fa01ae59008878c, entries=150, sequenceid=15, filesize=11.6 K 2023-04-28 14:58:29,577 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 1b6356a2d73db55249b3f0cb6bbd108a in 862ms, sequenceid=15, compaction requested=false 2023-04-28 14:58:29,577 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:29,667 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:29,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:29,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:29,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:29,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:29,669 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:29,669 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:29,669 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:29,669 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:29,670 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,670 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:29,670 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,670 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:29,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:29,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:29,670 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,670 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:29,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:29,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:29,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:29,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:29,671 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:29,671 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:29,671 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:29,671 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:29,671 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:29,671 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:29,671 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:29,671 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:29,671 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:29,672 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:29,672 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1b6356a2d73db55249b3f0cb6bbd108a 3/3 column families, dataSize=134.18 KB heapSize=366.23 KB 2023-04-28 14:58:29,672 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=A 2023-04-28 14:58:29,672 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:29,672 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=B 2023-04-28 14:58:29,672 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:29,672 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=C 2023-04-28 14:58:29,672 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:29,686 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042885a015f1fae44e19ab95e85bb20377a3 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042885a015f1fae44e19ab95e85bb20377a3 2023-04-28 14:58:29,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:29,845 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:29,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:29,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37450 deadline: 1682693969857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:29,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37454 deadline: 1682693969857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:29,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37496 deadline: 1682693969857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:29,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37424 deadline: 1682693969870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,875 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:29,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37478 deadline: 1682693969871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:29,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37454 deadline: 1682693969963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:29,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37450 deadline: 1682693969963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:29,968 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:29,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37496 deadline: 1682693969966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:30,091 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=48, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/2139eba17656443bb9439982b35222a4 2023-04-28 14:58:30,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:30,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37450 deadline: 1682693970167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:30,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:30,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37454 deadline: 1682693970167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:30,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:30,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37496 deadline: 1682693970169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:30,389 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-28 14:58:30,476 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:30,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37454 deadline: 1682693970473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:30,476 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:30,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37450 deadline: 1682693970473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:30,476 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:30,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37496 deadline: 1682693970474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:30,499 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/0aca0e5ea7744233b8271ba9f1ad98b6 2023-04-28 14:58:30,510 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/a230028051b94adfa5a13ef6f5b9052e 2023-04-28 14:58:30,515 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/2139eba17656443bb9439982b35222a4 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/2139eba17656443bb9439982b35222a4 2023-04-28 14:58:30,518 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/2139eba17656443bb9439982b35222a4, entries=150, sequenceid=48, filesize=25.1 K 2023-04-28 14:58:30,524 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/0aca0e5ea7744233b8271ba9f1ad98b6 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/0aca0e5ea7744233b8271ba9f1ad98b6 2023-04-28 14:58:30,528 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/0aca0e5ea7744233b8271ba9f1ad98b6, entries=150, sequenceid=48, filesize=11.6 K 2023-04-28 14:58:30,528 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/a230028051b94adfa5a13ef6f5b9052e as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/a230028051b94adfa5a13ef6f5b9052e 2023-04-28 14:58:30,534 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/a230028051b94adfa5a13ef6f5b9052e, entries=150, sequenceid=48, filesize=11.6 K 2023-04-28 14:58:30,534 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~366.19 KB/374976, currentSize=53.67 KB/54960 for 1b6356a2d73db55249b3f0cb6bbd108a in 862ms, sequenceid=48, compaction requested=false 2023-04-28 14:58:30,535 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:30,535 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:30,535 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:30,535 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:30,535 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:30,535 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:30,535 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:30,537 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:30,537 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:30,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:30,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:30,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:30,537 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:30,537 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:30,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:30,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:30,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:30,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:30,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:30,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:30,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a5cbfd[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:30,539 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:30,539 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:30,539 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:30,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:30,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:30,539 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:30,539 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:30,539 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:30,540 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:30,540 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:30,541 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:30,541 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:30,541 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:30,541 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:30,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:30,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:30,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:30,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:30,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:30,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:30,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:30,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:30,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:30,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:30,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:30,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:30,544 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:30,544 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:30,544 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:30,544 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:30,544 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:30,545 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:30,546 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:30,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:30,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:30,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:30,546 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:30,546 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:30,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:30,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:30,546 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:30,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:30,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:30,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:30,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:30,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:30,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:30,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:30,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:30,548 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:30,548 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:30,548 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:30,548 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:30,549 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:30,549 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:30,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:30,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:30,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:30,549 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:30,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:30,549 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:30,549 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:30,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:30,549 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:30,549 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:30,549 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:30,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:30,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:30,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:30,649 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:30,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:30,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:30,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:30,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:30,652 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:30,652 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:30,652 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:30,652 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:30,652 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:30,652 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:30,653 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:30,653 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:30,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:30,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:30,653 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:30,653 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:30,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:30,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:30,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:30,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:30,654 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:30,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:30,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:30,654 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:30,654 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:30,654 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:30,654 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:30,654 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:30,654 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:30,654 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:30,654 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1b6356a2d73db55249b3f0cb6bbd108a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-28 14:58:30,654 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=A 2023-04-28 14:58:30,654 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:30,654 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=B 2023-04-28 14:58:30,654 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:30,655 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=C 2023-04-28 14:58:30,655 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:30,667 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042898708376ae794ce3a08874348464d6ed to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042898708376ae794ce3a08874348464d6ed 2023-04-28 14:58:30,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:30,882 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:30,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693910907, max timestamp=1682693910924Num uniques 859; 2023-04-28 14:58:30,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693910907, max timestamp=1682693910924Num uniques 859; 2023-04-28 14:58:30,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:58:30,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693910907, max timestamp=1682693910924Num uniques 859; 2023-04-28 14:58:30,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:58:30,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:58:30,965 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:30,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37478 deadline: 1682693970958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:30,965 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:30,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37424 deadline: 1682693970959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:30,979 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:30,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37450 deadline: 1682693970977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:30,984 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:30,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37496 deadline: 1682693970979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:30,984 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:30,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37454 deadline: 1682693970980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,071 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=59, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/98f5ebaf27084bcdb60741015cf813ea 2023-04-28 14:58:31,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:31,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37478 deadline: 1682693971066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:31,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37424 deadline: 1682693971066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:31,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37424 deadline: 1682693971273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:31,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37478 deadline: 1682693971273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,479 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/76adf38f24534e2e85b1d5c370dcda3e 2023-04-28 14:58:31,488 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/684c5fd877d34127b5f57b210c1a1e91 2023-04-28 14:58:31,492 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/98f5ebaf27084bcdb60741015cf813ea as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/98f5ebaf27084bcdb60741015cf813ea 2023-04-28 14:58:31,495 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/98f5ebaf27084bcdb60741015cf813ea, entries=150, sequenceid=59, filesize=25.1 K 2023-04-28 14:58:31,495 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/76adf38f24534e2e85b1d5c370dcda3e as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/76adf38f24534e2e85b1d5c370dcda3e 2023-04-28 14:58:31,498 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/76adf38f24534e2e85b1d5c370dcda3e, entries=150, sequenceid=59, filesize=11.6 K 2023-04-28 14:58:31,499 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/684c5fd877d34127b5f57b210c1a1e91 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/684c5fd877d34127b5f57b210c1a1e91 2023-04-28 14:58:31,503 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/684c5fd877d34127b5f57b210c1a1e91, entries=150, sequenceid=59, filesize=11.6 K 2023-04-28 14:58:31,504 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 1b6356a2d73db55249b3f0cb6bbd108a in 850ms, sequenceid=59, compaction requested=true 2023-04-28 14:58:31,504 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:31,504 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:31,504 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:31,504 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:31,504 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,504 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:31,504 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:31,505 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,505 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:31,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:31,506 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,506 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:31,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:31,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:31,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:31,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:31,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:31,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b0e4df0[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:31,507 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:31,507 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,507 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,507 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:31,508 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:31,508 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:31,507 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,508 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:31,508 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:31,508 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:31,508 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,508 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:31,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:31,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:31,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:31,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:31,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:31,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:31,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:31,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:31,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:31,510 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:31,510 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:31,510 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:31,510 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:31,510 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:31,511 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:31,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:31,511 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,511 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:31,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:31,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:31,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:31,512 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:31,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:31,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:31,513 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:31,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:31,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,514 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,514 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:31,514 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:31,514 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,514 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,514 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,514 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:31,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:31,514 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:31,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:31,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:31,514 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:31,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:31,515 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:31,515 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:31,514 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:31,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:31,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:31,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:31,581 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1b6356a2d73db55249b3f0cb6bbd108a 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-28 14:58:31,582 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=A 2023-04-28 14:58:31,582 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:31,582 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=B 2023-04-28 14:58:31,582 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:31,582 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=C 2023-04-28 14:58:31,582 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:31,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:31,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37478 deadline: 1682693971603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:31,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37424 deadline: 1682693971604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,615 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:31,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:31,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:31,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:31,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:31,617 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:31,617 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:31,617 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:31,617 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:31,618 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,618 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:31,618 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,618 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:31,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:31,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:31,619 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,619 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:31,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:31,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:31,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:31,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:31,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:31,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:31,619 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:31,619 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:31,619 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:31,620 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:31,620 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:31,620 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:31,620 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:31,620 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:31,620 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:31,620 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:31,620 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:31,620 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,620 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:31,620 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:31,621 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,621 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:31,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:31,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:31,621 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,621 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:31,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:31,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:31,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:31,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:31,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@556ceef6[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:31,622 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:31,622 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,623 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,623 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:31,623 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,623 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:31,623 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:31,623 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:31,623 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,623 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:31,623 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:31,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:31,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:31,623 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:31,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:31,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:31,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:31,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:31,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:31,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:31,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:31,625 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:31,625 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:31,625 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:31,625 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:31,625 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:31,626 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,626 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:31,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,626 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:31,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:31,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:31,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:31,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:31,626 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:31,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:31,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:31,627 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:31,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:31,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,629 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,629 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:31,629 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:31,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:31,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:31,629 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:31,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:31,629 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:31,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:31,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:31,629 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:31,629 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,629 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:31,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:31,629 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:31,629 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:31,629 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,629 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:31,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37424 deadline: 1682693971713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:31,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37478 deadline: 1682693971713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,729 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:31,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:31,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:31,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:31,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:31,732 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:31,732 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:31,732 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:31,732 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:31,732 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,732 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:31,733 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,733 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:31,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:31,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:31,733 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,733 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:31,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:31,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:31,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:31,734 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:31,734 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:31,734 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:31,734 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:31,734 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:31,734 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:31,734 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:31,734 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:31,734 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:31,734 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:31,734 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:31,734 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:31,734 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:31,734 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:31,734 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,734 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:31,734 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:31,735 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,735 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:31,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:31,735 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,735 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:31,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:31,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:31,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:31,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:31,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:31,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@98b1e56[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:31,736 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:31,736 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,737 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,737 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:31,737 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:31,737 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:31,737 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,737 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:31,738 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,738 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:31,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:31,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:31,738 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:31,738 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:31,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:31,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:31,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:31,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:31,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:31,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:31,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:31,739 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:31,739 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:31,739 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:31,739 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:31,739 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:31,741 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,741 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,741 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:31,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:31,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:31,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,741 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:31,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:31,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:31,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:31,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:31,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:31,742 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:31,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:31,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,743 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,743 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:31,743 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:31,743 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:31,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:31,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:31,743 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:31,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:31,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:31,743 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:31,743 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,744 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:31,744 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:31,744 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,744 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,744 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:31,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:31,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:31,844 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:31,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:31,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:31,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:31,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:31,847 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:31,847 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:31,847 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:31,847 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:31,847 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,847 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:31,848 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:31,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:31,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:31,848 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:31,848 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,848 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:31,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:31,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:31,849 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:31,849 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:31,849 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:31,849 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:31,849 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:31,849 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:31,849 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:31,849 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:31,849 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:31,849 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:31,849 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:31,850 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:31,850 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:31,850 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:31,850 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,850 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:31,850 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:31,850 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:31,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:31,850 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:31,851 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,851 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:31,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:31,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:31,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:31,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:31,852 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:31,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26dec6b9[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:31,852 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,852 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,853 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:31,853 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:31,853 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:31,853 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,853 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:31,853 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,853 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:31,853 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:31,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:31,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:31,853 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:31,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:31,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:31,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:31,855 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:31,855 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:31,855 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:31,855 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:31,855 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:31,856 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:31,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:31,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:31,857 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:31,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:31,857 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,857 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:31,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:31,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,858 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:31,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:31,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:31,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:31,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,859 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,859 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:31,859 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:31,859 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:31,859 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:31,859 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:31,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:31,859 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:31,859 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:31,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:31,859 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:31,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:31,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:31,860 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,860 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:31,860 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:31,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:31,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37424 deadline: 1682693971923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:31,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37478 deadline: 1682693971923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,960 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:31,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:31,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:31,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:31,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:31,963 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:31,963 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:31,963 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:31,963 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:31,963 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,963 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:31,964 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:31,964 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:31,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:31,964 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,964 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:31,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:31,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:31,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:31,965 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:31,965 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:31,965 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:31,965 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:31,965 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:31,965 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:31,965 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:31,965 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:31,965 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:31,965 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:31,965 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:31,965 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:31,965 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:31,965 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:31,965 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,965 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:31,965 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:31,966 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,966 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:31,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:31,966 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,966 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:31,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:31,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:31,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:31,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:31,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:31,967 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:31,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fe67e5f[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:31,967 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,968 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,968 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:31,968 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:31,968 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:31,968 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,968 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:31,969 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:31,969 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:31,969 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,969 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:31,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:31,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:31,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:31,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:31,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:31,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:31,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:31,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:31,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:31,971 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:31,971 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:31,971 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:31,971 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:31,971 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:31,972 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,972 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:31,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:31,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,972 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:31,972 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:31,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:31,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,973 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:31,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:31,974 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,974 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:31,974 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:31,974 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:31,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:31,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:31,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:31,974 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:31,974 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:31,974 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:31,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:31,974 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,974 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,975 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:31,975 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:31,974 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:31,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:31,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:31,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:31,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:31,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:31,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:31,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37450 deadline: 1682693971981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,993 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042808fb6ff7e23340b6bff30eb8df776141 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042808fb6ff7e23340b6bff30eb8df776141 2023-04-28 14:58:31,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:31,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37454 deadline: 1682693971992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:31,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37496 deadline: 1682693971993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:31,999 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=96, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/1fe914c2bc8c4b53ba41f590ff5c3504 2023-04-28 14:58:32,075 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:32,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:32,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:32,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:32,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:32,078 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:32,078 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:32,078 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:32,078 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:32,078 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:32,078 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:32,079 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:32,079 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:32,079 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,079 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:32,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:32,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:32,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:32,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:32,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:32,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:32,080 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:32,080 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:32,080 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:32,080 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:32,080 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:32,080 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:32,080 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:32,081 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:32,081 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:32,081 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:32,081 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:32,081 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,081 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:32,081 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:32,081 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,081 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:32,082 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:32,082 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:32,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:32,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:32,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dcf240b[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:32,083 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:32,083 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:32,084 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:32,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:32,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:32,084 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:32,084 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:32,084 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:32,084 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,084 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:32,084 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:32,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:32,084 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:32,085 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:32,085 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:32,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:32,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:32,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:32,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:32,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:32,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:32,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:32,086 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:32,086 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:32,086 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:32,086 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:32,087 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:32,087 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:32,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:32,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:32,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:32,087 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:32,087 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:32,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:32,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:32,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:32,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:32,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:32,088 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:32,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:32,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:32,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:32,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,089 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:32,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:32,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,091 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,091 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:32,091 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:32,091 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,091 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:32,091 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:32,091 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:32,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:32,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:32,091 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:32,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:32,091 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:32,091 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:32,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:32,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:32,091 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:32,091 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:32,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:32,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:32,191 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:32,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:32,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:32,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:32,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:32,194 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:32,194 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:32,194 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:32,194 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:32,194 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:32,194 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:32,195 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,195 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:32,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:32,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:32,195 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,195 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:32,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:32,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:32,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:32,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:32,196 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:32,196 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:32,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:32,196 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:32,196 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:32,196 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:32,196 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:32,196 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:32,196 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:32,196 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:32,197 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:32,197 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:32,197 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:32,197 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:32,197 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,197 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:32,197 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:32,198 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:32,198 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:32,198 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:32,198 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:32,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:32,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:32,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:32,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:32,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:32,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ce7ea4a[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:32,200 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:32,200 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:32,201 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:32,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:32,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:32,201 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:32,201 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:32,201 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:32,201 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,201 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:32,202 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:32,202 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,202 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:32,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:32,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:32,202 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:32,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:32,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:32,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:32,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:32,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:32,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:32,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:32,204 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:32,204 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:32,204 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:32,204 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:32,204 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:32,205 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:32,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:32,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:32,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:32,205 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:32,205 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:32,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:32,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:32,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:32,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:32,205 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:32,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:32,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:32,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:32,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:32,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,206 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:32,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:32,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:32,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:32,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,208 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:32,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:32,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:32,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:32,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:32,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:32,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:32,208 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:32,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:32,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:32,208 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:32,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:32,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:32,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:32,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:32,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37424 deadline: 1682693972230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,235 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:32,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37478 deadline: 1682693972231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,308 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:32,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:32,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:32,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:32,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:32,311 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:32,311 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:32,311 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:32,311 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:32,312 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:32,312 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:32,312 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,312 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:32,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:32,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:32,312 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,312 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:32,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:32,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:32,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:32,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:32,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:32,314 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:32,314 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:32,314 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:32,314 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:32,314 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:32,314 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:32,314 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:32,314 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:32,314 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:32,315 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:32,315 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:32,315 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:32,315 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:32,315 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,315 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:32,315 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:32,315 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,315 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:32,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:32,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:32,316 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:32,316 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:32,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:32,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:32,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:32,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:32,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:32,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c5c1c61[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:32,317 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:32,317 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:32,322 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:32,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:32,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:32,322 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:32,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:32,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:32,323 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,323 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:32,323 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:32,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:32,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:32,324 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:32,324 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:32,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:32,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:32,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:32,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:32,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:32,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:32,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:32,327 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:32,328 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:32,328 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:32,328 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:32,328 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:32,329 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:32,329 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:32,329 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:32,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:32,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:32,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:32,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:32,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:32,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:32,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:32,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:32,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:32,330 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:32,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:32,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:32,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,332 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:32,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:32,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,333 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,333 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:32,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:32,334 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:32,333 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:32,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:32,334 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:32,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:32,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:32,334 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:32,334 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:32,334 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:32,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:32,334 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:32,334 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:32,334 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:32,334 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:32,407 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/74201117f358422a9bc72df0fede463a 2023-04-28 14:58:32,417 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/2d50b616bc2b41d1bf2836b544db4943 2023-04-28 14:58:32,420 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/1fe914c2bc8c4b53ba41f590ff5c3504 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/1fe914c2bc8c4b53ba41f590ff5c3504 2023-04-28 14:58:32,423 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/1fe914c2bc8c4b53ba41f590ff5c3504, entries=200, sequenceid=96, filesize=31.9 K 2023-04-28 14:58:32,423 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/74201117f358422a9bc72df0fede463a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/74201117f358422a9bc72df0fede463a 2023-04-28 14:58:32,426 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/74201117f358422a9bc72df0fede463a, entries=150, sequenceid=96, filesize=11.6 K 2023-04-28 14:58:32,427 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/2d50b616bc2b41d1bf2836b544db4943 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/2d50b616bc2b41d1bf2836b544db4943 2023-04-28 14:58:32,430 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/2d50b616bc2b41d1bf2836b544db4943, entries=150, sequenceid=96, filesize=11.6 K 2023-04-28 14:58:32,430 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 1b6356a2d73db55249b3f0cb6bbd108a in 849ms, sequenceid=96, compaction requested=true 2023-04-28 14:58:32,430 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:32,431 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:32,431 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:58:32,431 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:58:32,431 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:32,431 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:32,433 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 116864 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:58:32,433 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): 1b6356a2d73db55249b3f0cb6bbd108a/A is initiating minor compaction (all files) 2023-04-28 14:58:32,433 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:58:32,433 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1b6356a2d73db55249b3f0cb6bbd108a/A in TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:32,434 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 1b6356a2d73db55249b3f0cb6bbd108a/B is initiating minor compaction (all files) 2023-04-28 14:58:32,434 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/8cc291db9d1e4b5aaf853dc2ebbb8272, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/2139eba17656443bb9439982b35222a4, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/98f5ebaf27084bcdb60741015cf813ea, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/1fe914c2bc8c4b53ba41f590ff5c3504] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp, totalSize=114.1 K 2023-04-28 14:58:32,434 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1b6356a2d73db55249b3f0cb6bbd108a/B in TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:32,434 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/f47036c560f64891946b5871cfadbc26, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/0aca0e5ea7744233b8271ba9f1ad98b6, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/76adf38f24534e2e85b1d5c370dcda3e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/74201117f358422a9bc72df0fede463a] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp, totalSize=46.4 K 2023-04-28 14:58:32,434 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 8cc291db9d1e4b5aaf853dc2ebbb8272, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682693908708 2023-04-28 14:58:32,434 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting f47036c560f64891946b5871cfadbc26, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682693908713 2023-04-28 14:58:32,434 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 2139eba17656443bb9439982b35222a4, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682693908735 2023-04-28 14:58:32,434 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 0aca0e5ea7744233b8271ba9f1ad98b6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682693908735 2023-04-28 14:58:32,434 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:32,434 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 98f5ebaf27084bcdb60741015cf813ea, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682693909856 2023-04-28 14:58:32,434 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 76adf38f24534e2e85b1d5c370dcda3e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682693909856 2023-04-28 14:58:32,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:32,435 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 1fe914c2bc8c4b53ba41f590ff5c3504, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1682693910950 2023-04-28 14:58:32,435 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 74201117f358422a9bc72df0fede463a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1682693910956 2023-04-28 14:58:32,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:32,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:32,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:32,437 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:32,437 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:32,437 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:32,437 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:32,438 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:32,438 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:32,438 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,438 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:32,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:32,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:32,438 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,438 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:32,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:32,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:32,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:32,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:32,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:32,441 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:32,441 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:32,441 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:32,441 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:32,441 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:32,441 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:32,441 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:32,441 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:32,441 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:32,441 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1b6356a2d73db55249b3f0cb6bbd108a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 14:58:32,442 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=A 2023-04-28 14:58:32,442 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:32,442 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=B 2023-04-28 14:58:32,442 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:32,442 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=C 2023-04-28 14:58:32,442 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:32,445 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 1b6356a2d73db55249b3f0cb6bbd108a#B#compaction#294 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:32,459 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1b6356a2d73db55249b3f0cb6bbd108a#A#compaction#295 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:32,465 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/7139dae4f3a847f6983d39f5795f45a7 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/7139dae4f3a847f6983d39f5795f45a7 2023-04-28 14:58:32,468 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1b6356a2d73db55249b3f0cb6bbd108a/B of 1b6356a2d73db55249b3f0cb6bbd108a into 7139dae4f3a847f6983d39f5795f45a7(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:32,468 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:32,468 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a., storeName=1b6356a2d73db55249b3f0cb6bbd108a/B, priority=12, startTime=1682693912431; duration=0sec 2023-04-28 14:58:32,468 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:32,468 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:58:32,469 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:58:32,469 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 1b6356a2d73db55249b3f0cb6bbd108a/C is initiating minor compaction (all files) 2023-04-28 14:58:32,469 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1b6356a2d73db55249b3f0cb6bbd108a/C in TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:32,469 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/b70b8d11e45945879fa01ae59008878c, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/a230028051b94adfa5a13ef6f5b9052e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/684c5fd877d34127b5f57b210c1a1e91, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/2d50b616bc2b41d1bf2836b544db4943] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp, totalSize=46.4 K 2023-04-28 14:58:32,470 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting b70b8d11e45945879fa01ae59008878c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682693908713 2023-04-28 14:58:32,470 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting a230028051b94adfa5a13ef6f5b9052e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682693908735 2023-04-28 14:58:32,470 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 684c5fd877d34127b5f57b210c1a1e91, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682693909856 2023-04-28 14:58:32,470 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 2d50b616bc2b41d1bf2836b544db4943, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1682693910956 2023-04-28 14:58:32,480 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 1b6356a2d73db55249b3f0cb6bbd108a#C#compaction#297 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:32,487 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/17b70c903b714e83a2fdab80eaa83098 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/17b70c903b714e83a2fdab80eaa83098 2023-04-28 14:58:32,490 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1b6356a2d73db55249b3f0cb6bbd108a/C of 1b6356a2d73db55249b3f0cb6bbd108a into 17b70c903b714e83a2fdab80eaa83098(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:32,490 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:32,490 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a., storeName=1b6356a2d73db55249b3f0cb6bbd108a/C, priority=12, startTime=1682693912431; duration=0sec 2023-04-28 14:58:32,490 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:32,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:32,735 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:32,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682693912756, max timestamp=1682693912772Num uniques 859; 2023-04-28 14:58:32,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682693912756, max timestamp=1682693912772Num uniques 859; 2023-04-28 14:58:32,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682693912756, max timestamp=1682693912772Num uniques 859; 2023-04-28 14:58:32,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 14:58:32,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 14:58:32,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 14:58:32,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:32,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37478 deadline: 1682693972806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:32,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37424 deadline: 1682693972806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,866 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428c3a26e6d8e124eaabceba5b74cab9678 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428c3a26e6d8e124eaabceba5b74cab9678 2023-04-28 14:58:32,917 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:32,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37478 deadline: 1682693972913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:32,918 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:32,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37424 deadline: 1682693972913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:33,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:33,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37478 deadline: 1682693973119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:33,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:33,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37424 deadline: 1682693973119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:33,278 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=106, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/1e5211d6c9be47d2af90f35ae1d19606 2023-04-28 14:58:33,287 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/2a9b9c1d30a641ae8df30685f492177d 2023-04-28 14:58:33,288 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/904d6b2d9cf44200a3f6a70cee5ec6e1 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/904d6b2d9cf44200a3f6a70cee5ec6e1 2023-04-28 14:58:33,292 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1b6356a2d73db55249b3f0cb6bbd108a/A of 1b6356a2d73db55249b3f0cb6bbd108a into 904d6b2d9cf44200a3f6a70cee5ec6e1(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:33,292 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:33,292 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a., storeName=1b6356a2d73db55249b3f0cb6bbd108a/A, priority=12, startTime=1682693912431; duration=0sec 2023-04-28 14:58:33,292 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:33,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:33,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37478 deadline: 1682693973421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:33,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:33,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37424 deadline: 1682693973421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:33,696 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/52fd3a4b770b4489b78a838d85796a05 2023-04-28 14:58:33,699 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/1e5211d6c9be47d2af90f35ae1d19606 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/1e5211d6c9be47d2af90f35ae1d19606 2023-04-28 14:58:33,702 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/1e5211d6c9be47d2af90f35ae1d19606, entries=150, sequenceid=106, filesize=25.1 K 2023-04-28 14:58:33,703 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/2a9b9c1d30a641ae8df30685f492177d as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/2a9b9c1d30a641ae8df30685f492177d 2023-04-28 14:58:33,706 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/2a9b9c1d30a641ae8df30685f492177d, entries=150, sequenceid=106, filesize=11.6 K 2023-04-28 14:58:33,706 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/52fd3a4b770b4489b78a838d85796a05 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/52fd3a4b770b4489b78a838d85796a05 2023-04-28 14:58:33,709 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/52fd3a4b770b4489b78a838d85796a05, entries=150, sequenceid=106, filesize=11.6 K 2023-04-28 14:58:33,710 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 1b6356a2d73db55249b3f0cb6bbd108a in 1269ms, sequenceid=106, compaction requested=false 2023-04-28 14:58:33,710 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:33,710 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:33,710 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:33,710 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:33,710 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:33,711 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:33,711 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:33,712 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:33,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:33,712 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:33,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:33,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:33,712 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:33,712 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:33,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:33,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:33,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:33,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:33,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:33,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:33,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@849733[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:33,714 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:33,714 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:33,714 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:33,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:33,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:33,714 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:33,714 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:33,714 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:33,714 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:33,714 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:33,715 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:33,715 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:33,715 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:33,715 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:33,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:33,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:33,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:33,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:33,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:33,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:33,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:33,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:33,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:33,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:33,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:33,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:33,717 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:33,717 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:33,717 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:33,717 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:33,717 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:33,718 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:33,718 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:33,718 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:33,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:33,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:33,719 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:33,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:33,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:33,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:33,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:33,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:33,720 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:33,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:33,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:33,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:33,721 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:33,721 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:33,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:33,722 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:33,721 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:33,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:33,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:33,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:33,722 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:33,722 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:33,722 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:33,722 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:33,722 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:33,722 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:33,722 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:33,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:33,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:33,722 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:33,822 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:33,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:33,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:33,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:33,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:33,825 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:33,825 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:33,825 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:33,825 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:33,825 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:33,825 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:33,826 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:33,826 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:33,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:33,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:33,826 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:33,826 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:33,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:33,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:33,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:33,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:33,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:33,827 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:33,827 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:33,827 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:33,827 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:33,827 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:33,827 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:33,827 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:33,827 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:33,827 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:33,828 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1b6356a2d73db55249b3f0cb6bbd108a 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-28 14:58:33,828 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=A 2023-04-28 14:58:33,828 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:33,828 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=B 2023-04-28 14:58:33,828 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:33,828 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=C 2023-04-28 14:58:33,828 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:33,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:33,929 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:33,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:33,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37424 deadline: 1682693973945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:33,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:33,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37478 deadline: 1682693973949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:34,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:34,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37454 deadline: 1682693974005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:34,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:34,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37450 deadline: 1682693974005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:34,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:34,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37496 deadline: 1682693974006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:34,011 DEBUG [Thread-2107] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a., hostname=jenkins-hbase17.apache.org,37181,1682693812482, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 14:58:34,012 DEBUG [Thread-2105] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a., hostname=jenkins-hbase17.apache.org,37181,1682693812482, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 14:58:34,013 DEBUG [Thread-2103] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4155 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a., hostname=jenkins-hbase17.apache.org,37181,1682693812482, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 14:58:34,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:34,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37424 deadline: 1682693974050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:34,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:34,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37478 deadline: 1682693974056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:34,239 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428de824d32214d4b52be7cc10f18b27b46 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428de824d32214d4b52be7cc10f18b27b46 2023-04-28 14:58:34,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:34,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37424 deadline: 1682693974254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:34,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:34,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37478 deadline: 1682693974261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:34,568 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:34,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37424 deadline: 1682693974561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:34,571 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:34,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37478 deadline: 1682693974567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:34,644 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=144, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/ee9b85658f1943e7b84b049563af6afe 2023-04-28 14:58:35,053 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/e947743dfcbf4590a7a29276fae44ea4 2023-04-28 14:58:35,073 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:35,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37424 deadline: 1682693975070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:35,079 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:35,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37478 deadline: 1682693975076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:35,461 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/cf3ba797893847148b398d7d16af6573 2023-04-28 14:58:35,464 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/ee9b85658f1943e7b84b049563af6afe as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/ee9b85658f1943e7b84b049563af6afe 2023-04-28 14:58:35,467 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/ee9b85658f1943e7b84b049563af6afe, entries=150, sequenceid=144, filesize=25.3 K 2023-04-28 14:58:35,468 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/e947743dfcbf4590a7a29276fae44ea4 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/e947743dfcbf4590a7a29276fae44ea4 2023-04-28 14:58:35,471 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/e947743dfcbf4590a7a29276fae44ea4, entries=150, sequenceid=144, filesize=11.8 K 2023-04-28 14:58:35,472 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/cf3ba797893847148b398d7d16af6573 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/cf3ba797893847148b398d7d16af6573 2023-04-28 14:58:35,475 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/cf3ba797893847148b398d7d16af6573, entries=150, sequenceid=144, filesize=11.8 K 2023-04-28 14:58:35,476 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 1b6356a2d73db55249b3f0cb6bbd108a in 1648ms, sequenceid=144, compaction requested=true 2023-04-28 14:58:35,476 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:35,476 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:35,476 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:35,476 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:35,476 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:35,476 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:35,476 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:35,477 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:35,477 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:35,478 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:35,478 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:35,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:35,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:35,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:35,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:35,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:35,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:35,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34df0454[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:35,479 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:35,479 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:35,479 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:35,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:35,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:35,480 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:35,480 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:35,480 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:35,480 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:35,480 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:35,480 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:35,480 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:35,480 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:35,480 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:35,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:35,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:35,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:35,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:35,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:35,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:35,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:35,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:35,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:35,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:35,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:35,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:35,482 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:35,482 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:35,482 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:35,482 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:35,482 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:35,483 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:35,483 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:35,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:35,483 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:35,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:35,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:35,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:35,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:35,484 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:35,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:35,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:35,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:35,485 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:35,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:35,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:35,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:35,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:35,488 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:35,488 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:35,488 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:35,488 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:35,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:35,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:35,489 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:35,489 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:35,489 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:35,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:35,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:35,489 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:35,489 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:35,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:35,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:35,489 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:35,490 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:35,490 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:35,590 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:35,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:35,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:35,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:35,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:35,593 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:35,593 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:35,593 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:35,593 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:35,593 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:35,593 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:35,593 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:35,594 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:35,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:35,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:35,594 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:35,594 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:35,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:35,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:35,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:35,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:35,594 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:35,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:35,595 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:35,595 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:35,595 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:35,595 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:35,595 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:35,595 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:35,595 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:35,595 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:35,595 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1b6356a2d73db55249b3f0cb6bbd108a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 14:58:35,595 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=A 2023-04-28 14:58:35,595 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:35,595 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=B 2023-04-28 14:58:35,595 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:35,596 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=C 2023-04-28 14:58:35,596 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:36,006 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428f4648bf067384a32ab0dad90d3a3a485 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428f4648bf067384a32ab0dad90d3a3a485 2023-04-28 14:58:36,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:36,087 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:36,139 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693916130, max timestamp=1682693916137Num uniques 859; 2023-04-28 14:58:36,139 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693916130, max timestamp=1682693916137Num uniques 859; 2023-04-28 14:58:36,139 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:58:36,139 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693916130, max timestamp=1682693916137Num uniques 859; 2023-04-28 14:58:36,139 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:58:36,139 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:58:36,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:36,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37424 deadline: 1682693976196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:36,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:36,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37478 deadline: 1682693976197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:36,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:36,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37424 deadline: 1682693976302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:36,311 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:36,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37478 deadline: 1682693976304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:36,411 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=154, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/d45bf5f28f9f4c299e1d92de9c65c639 2023-04-28 14:58:36,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:36,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37424 deadline: 1682693976506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:36,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:36,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37478 deadline: 1682693976512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:36,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:36,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37424 deadline: 1682693976812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:36,820 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/50e702134ae74dd6884303929eb10233 2023-04-28 14:58:36,821 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:36,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37478 deadline: 1682693976815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,229 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/f660052b0d1045f7893cdfcd2d5ee1b4 2023-04-28 14:58:37,232 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/d45bf5f28f9f4c299e1d92de9c65c639 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/d45bf5f28f9f4c299e1d92de9c65c639 2023-04-28 14:58:37,236 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/d45bf5f28f9f4c299e1d92de9c65c639, entries=150, sequenceid=154, filesize=25.3 K 2023-04-28 14:58:37,236 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/50e702134ae74dd6884303929eb10233 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/50e702134ae74dd6884303929eb10233 2023-04-28 14:58:37,239 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/50e702134ae74dd6884303929eb10233, entries=150, sequenceid=154, filesize=11.8 K 2023-04-28 14:58:37,240 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/f660052b0d1045f7893cdfcd2d5ee1b4 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/f660052b0d1045f7893cdfcd2d5ee1b4 2023-04-28 14:58:37,242 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/f660052b0d1045f7893cdfcd2d5ee1b4, entries=150, sequenceid=154, filesize=11.8 K 2023-04-28 14:58:37,243 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 1b6356a2d73db55249b3f0cb6bbd108a in 1648ms, sequenceid=154, compaction requested=true 2023-04-28 14:58:37,243 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:37,243 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:37,243 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:37,243 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:37,243 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,243 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:37,243 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:37,244 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,244 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:37,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:37,244 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,244 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:37,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:37,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:37,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:37,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:37,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17c91d3f[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:37,245 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:37,245 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,246 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,246 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:37,246 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:37,246 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:37,246 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,246 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:37,246 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:37,246 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,246 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:37,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:37,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:37,247 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:37,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:37,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:37,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:37,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:37,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:37,248 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:37,248 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:37,248 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:37,248 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:37,248 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:37,249 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,249 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:37,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:37,249 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:37,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:37,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:37,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:37,250 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:37,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,250 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:37,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,252 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,252 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,252 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:37,252 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:37,252 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:37,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:37,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:37,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:37,252 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:37,252 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,252 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:37,252 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:37,252 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:37,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:37,252 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,252 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:37,327 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1b6356a2d73db55249b3f0cb6bbd108a 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-28 14:58:37,328 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=A 2023-04-28 14:58:37,328 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 14:58:37,328 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=B 2023-04-28 14:58:37,328 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 14:58:37,328 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=C 2023-04-28 14:58:37,328 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 14:58:37,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:37,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37478 deadline: 1682693977346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,352 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:37,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:37,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:37,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:37,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:37,355 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:37,355 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:37,355 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:37,355 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:37,355 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,355 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,356 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:37,356 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:37,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:37,356 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,356 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:37,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:37,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:37,357 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:37,357 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:37,357 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:37,357 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:37,357 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:37,357 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:37,357 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:37,357 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:37,357 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:37,357 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:37,357 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:37,357 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:37,357 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:37,357 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,357 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:37,357 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:37,358 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,358 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:37,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:37,358 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,358 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:37,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:37,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:37,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:37,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:37,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b5ab4b3[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:37,359 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:37,359 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,360 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:37,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693977347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,361 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,361 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:37,361 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,361 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:37,361 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:37,361 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:37,362 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,362 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:37,362 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:37,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:37,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:37,362 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:37,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:37,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:37,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:37,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:37,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:37,364 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:37,364 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:37,364 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:37,364 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:37,364 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:37,365 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,365 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:37,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:37,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,365 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:37,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:37,365 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:37,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,366 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:37,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,368 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,368 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,368 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:37,368 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,368 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:37,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:37,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:37,368 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,368 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,368 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,368 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,368 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:37,368 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:37,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:37,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:37,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:37,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:37,457 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:37,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37478 deadline: 1682693977452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:37,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693977460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,468 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:37,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:37,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:37,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:37,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:37,470 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:37,471 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:37,471 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:37,471 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:37,471 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,471 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,471 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,471 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:37,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:37,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:37,472 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,472 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:37,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:37,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,472 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:37,473 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:37,473 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:37,473 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:37,473 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:37,473 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:37,473 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:37,473 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:37,473 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:37,473 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:37,473 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:37,473 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:37,473 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:37,473 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:37,473 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,473 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:37,473 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:37,475 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,475 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:37,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:37,476 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:37,476 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:37,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:37,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:37,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:37,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f4381e4[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:37,477 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:37,477 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,477 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,478 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:37,478 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:37,478 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:37,478 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,478 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:37,478 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,478 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:37,478 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:37,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:37,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:37,478 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:37,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:37,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:37,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:37,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:37,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:37,480 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:37,480 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:37,480 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:37,480 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:37,480 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:37,481 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,481 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:37,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:37,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,481 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:37,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:37,481 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:37,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:37,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:37,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,482 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,484 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,484 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,484 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:37,484 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,484 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,484 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:37,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:37,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:37,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:37,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:37,484 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,484 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,484 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,484 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:37,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:37,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:37,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:37,484 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:37,484 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:37,585 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:37,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:37,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:37,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:37,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:37,587 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:37,587 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:37,587 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:37,587 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:37,588 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,588 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,588 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,588 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:37,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:37,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:37,588 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,589 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:37,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:37,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:37,589 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:37,589 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:37,589 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:37,590 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:37,590 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:37,590 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:37,590 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:37,590 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:37,590 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:37,590 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:37,590 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:37,590 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:37,590 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:37,590 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,590 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:37,590 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:37,591 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,591 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:37,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:37,591 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:37,591 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:37,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:37,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:37,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:37,592 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:37,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53f2dbfc[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:37,592 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,593 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,593 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:37,593 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,593 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:37,593 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:37,593 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:37,594 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,594 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:37,594 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:37,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:37,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:37,594 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:37,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:37,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:37,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:37,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:37,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:37,596 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:37,596 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:37,596 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:37,596 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:37,596 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:37,596 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,597 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:37,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:37,597 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:37,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:37,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:37,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:37,597 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:37,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,598 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:37,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,601 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,601 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,601 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:37,601 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:37,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:37,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:37,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:37,601 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:37,601 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:37,601 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,601 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,601 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,601 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:37,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:37,601 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:37,601 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:37,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:37,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37478 deadline: 1682693977659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:37,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693977665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,701 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:37,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:37,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:37,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:37,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:37,704 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:37,704 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:37,704 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:37,704 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:37,705 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,705 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,705 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,705 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:37,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:37,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:37,706 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,706 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:37,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:37,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,706 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:37,706 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:37,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:37,706 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:37,706 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:37,706 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:37,707 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:37,707 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:37,707 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:37,707 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:37,707 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:37,707 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:37,707 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:37,707 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:37,707 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,707 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:37,707 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:37,708 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:37,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:37,708 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:37,708 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,708 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:37,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:37,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:37,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:37,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c8d1458[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:37,709 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:37,709 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,710 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,710 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:37,710 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:37,710 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:37,710 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,710 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:37,711 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:37,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:37,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:37,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:37,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,711 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:37,712 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:37,712 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:37,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:37,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:37,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:37,713 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:37,713 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:37,713 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:37,713 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:37,713 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:37,713 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,713 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:37,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:37,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,713 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:37,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:37,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:37,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:37,714 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:37,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,715 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:37,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,716 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,716 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,716 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,716 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:37,716 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:37,716 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:37,716 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:37,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:37,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:37,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:37,716 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,717 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,717 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,717 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:37,717 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:37,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:37,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:37,738 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428adb2f4d4065e469d886d2762c701ff9f to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428adb2f4d4065e469d886d2762c701ff9f 2023-04-28 14:58:37,817 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:37,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:37,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:37,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:37,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:37,820 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:37,820 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:37,820 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:37,820 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:37,820 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,820 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,821 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:37,821 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:37,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:37,821 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,821 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:37,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:37,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,822 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:37,822 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:37,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:37,823 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:37,823 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:37,823 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:37,823 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:37,823 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:37,823 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:37,823 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:37,823 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:37,823 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:37,823 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:37,823 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:37,823 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,823 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:37,823 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:37,824 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:37,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:37,824 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:37,824 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,825 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:37,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:37,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:37,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:37,825 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:37,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@629cc656[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:37,825 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,827 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,827 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:37,827 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,827 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:37,827 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:37,827 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:37,828 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,828 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:37,828 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:37,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:37,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:37,828 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:37,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:37,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:37,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:37,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:37,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:37,829 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:37,829 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:37,829 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:37,829 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:37,830 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:37,830 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:37,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:37,830 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,830 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:37,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:37,831 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:37,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:37,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:37,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,832 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:37,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,833 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,833 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:37,833 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:37,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:37,833 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:37,833 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,833 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,833 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,833 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:37,833 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:37,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:37,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:37,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:37,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:37,834 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:37,833 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,834 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:37,934 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:37,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:37,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:37,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:37,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:37,936 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:37,936 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:37,936 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:37,937 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:37,937 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,937 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,937 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:37,937 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:37,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:37,938 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,938 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:37,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:37,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,938 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:37,939 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:37,939 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:37,939 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:37,939 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:37,939 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:37,939 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:37,939 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:37,939 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:37,939 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:37,939 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:37,939 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:37,939 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:37,939 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:37,939 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,939 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:37,939 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:37,940 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,940 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:37,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:37,940 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,940 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:37,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:37,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:37,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:37,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:37,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61d33740[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:37,942 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:37,942 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,942 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,942 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:37,942 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:37,943 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:37,943 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,943 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:37,943 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:37,943 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,943 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:37,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:37,943 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:37,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:37,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:37,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:37,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:37,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:37,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:37,945 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:37,945 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:37,945 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:37,945 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:37,945 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:37,946 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,946 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:37,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:37,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,946 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:37,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:37,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:37,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:37,947 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:37,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,948 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:37,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:37,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,949 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,949 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,949 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:37,949 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:37,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:37,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:37,949 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:37,949 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:37,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:37,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:37,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:37,949 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:37,949 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,950 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:37,950 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:37,950 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:37,950 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:37,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:37,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37478 deadline: 1682693977965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:37,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:37,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693977970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,028 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:38,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37454 deadline: 1682693978024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,030 DEBUG [Thread-2107] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8171 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a., hostname=jenkins-hbase17.apache.org,37181,1682693812482, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 14:58:38,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:38,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37496 deadline: 1682693978038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,045 DEBUG [Thread-2103] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8186 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a., hostname=jenkins-hbase17.apache.org,37181,1682693812482, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 14:58:38,050 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:38,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:38,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:38,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:38,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37450 deadline: 1682693978046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:38,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:38,053 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:38,053 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:38,053 DEBUG [Thread-2105] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8195 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a., hostname=jenkins-hbase17.apache.org,37181,1682693812482, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 14:58:38,053 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:38,054 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:38,054 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,054 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,055 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,055 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:38,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:38,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:38,055 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,055 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:38,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:38,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,056 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:38,056 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:38,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:38,056 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:38,056 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:38,056 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:38,056 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:38,056 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:38,056 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:38,056 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:38,057 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:38,057 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:38,057 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:38,057 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:38,057 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,057 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:38,057 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:38,058 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,058 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:38,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:38,058 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,058 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:38,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:38,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:38,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:38,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:38,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@243b3dbd[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:38,061 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:38,061 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,061 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,062 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:38,062 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:38,062 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,062 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:38,062 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:38,062 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:38,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:38,062 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:38,063 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:38,063 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:38,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:38,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:38,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:38,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:38,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:38,064 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:38,064 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:38,064 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:38,064 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:38,064 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:38,065 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,065 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:38,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:38,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,065 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:38,066 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:38,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:38,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:38,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:38,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,067 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:38,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,069 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,069 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:38,069 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:38,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:38,069 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,069 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:38,069 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:38,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:38,069 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:38,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:38,069 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,070 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,070 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:38,070 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:38,070 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:38,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:38,143 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=192, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/3bb39b3d938f45bca45ed1c7c5c33b1b 2023-04-28 14:58:38,170 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:38,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:38,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:38,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:38,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:38,172 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:38,172 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:38,173 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:38,173 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:38,173 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,173 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,174 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:38,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:38,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:38,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:38,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:38,175 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,175 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:38,175 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:38,175 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:38,175 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:38,175 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,175 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:38,175 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:38,176 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:38,176 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:38,176 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:38,176 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:38,176 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:38,176 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:38,176 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:38,176 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:38,176 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,176 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:38,176 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:38,177 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:38,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:38,177 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:38,177 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,177 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:38,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:38,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:38,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:38,180 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:38,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4070c2b6[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:38,180 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,180 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,180 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,180 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:38,180 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:38,181 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:38,181 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:38,182 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:38,182 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,182 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:38,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:38,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:38,182 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:38,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:38,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:38,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:38,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:38,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:38,184 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:38,184 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:38,184 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:38,184 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:38,184 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:38,185 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,185 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,185 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:38,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:38,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:38,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:38,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:38,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:38,185 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:38,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,186 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:38,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,188 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,188 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:38,188 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,188 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:38,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:38,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:38,188 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,188 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,188 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,188 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,188 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,188 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:38,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:38,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:38,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:38,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:38,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:38,189 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:38,189 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:38,289 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:38,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:38,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:38,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:38,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:38,292 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:38,292 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:38,292 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:38,292 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:38,292 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,292 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,292 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,293 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:38,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:38,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:38,293 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,293 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:38,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:38,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,293 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:38,294 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:38,294 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:38,294 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:38,294 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:38,294 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:38,294 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:38,294 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:38,294 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:38,294 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:38,294 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:38,294 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:38,295 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:38,295 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:38,295 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,295 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:38,295 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:38,295 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:38,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:38,296 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:38,296 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,296 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:38,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:38,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:38,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:38,297 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:38,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44da1298[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:38,297 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,297 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,298 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:38,298 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:38,298 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,298 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:38,298 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:38,298 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:38,298 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:38,298 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,298 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:38,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:38,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:38,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:38,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:38,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:38,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:38,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:38,300 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:38,300 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:38,300 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:38,300 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:38,300 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:38,301 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:38,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:38,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:38,302 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:38,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,302 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:38,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,303 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,303 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:38,304 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,304 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:38,304 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,304 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,304 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,304 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,304 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:38,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:38,305 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:38,305 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:38,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,304 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,304 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:38,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:38,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:38,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:38,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:38,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:38,405 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:38,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:38,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:38,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:38,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:38,408 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:38,408 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:38,408 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:38,408 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:38,408 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,408 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,409 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:38,409 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:38,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:38,409 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,409 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:38,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:38,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,410 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:38,410 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:38,410 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:38,410 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:38,410 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:38,410 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:38,410 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:38,410 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:38,410 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:38,410 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:38,410 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:38,410 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:38,410 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:38,410 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:38,410 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,411 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:38,411 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:38,411 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,411 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:38,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:38,411 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,411 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:38,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:38,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:38,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:38,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:38,413 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:38,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ccb4474[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:38,413 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,413 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,413 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:38,413 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:38,413 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:38,414 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,414 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:38,414 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:38,414 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:38,414 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,414 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:38,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:38,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:38,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:38,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:38,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:38,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:38,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:38,417 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:38,417 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:38,417 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:38,417 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:38,417 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:38,418 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,418 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:38,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:38,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:38,418 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:38,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:38,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,419 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:38,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:38,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,420 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:38,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,422 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,422 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,422 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:38,422 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,422 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,422 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:38,422 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:38,422 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:38,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:38,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:38,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:38,422 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:38,422 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,422 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:38,422 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:38,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:38,479 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:38,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37478 deadline: 1682693978473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:38,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693978478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,522 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:38,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:38,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:38,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:38,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:38,529 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:38,529 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:38,529 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:38,529 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:38,530 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,530 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,530 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:38,530 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:38,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:38,531 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,531 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:38,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:38,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,531 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:38,531 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:38,531 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:38,531 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:38,531 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:38,532 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:38,532 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:38,532 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:38,532 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:38,532 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:38,532 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:38,532 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:38,532 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:38,532 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:38,532 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,532 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:38,532 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:38,533 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,533 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:38,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:38,533 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,533 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:38,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:38,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:38,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:38,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:38,534 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:38,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41e04085[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:38,534 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,535 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,535 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:38,535 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:38,535 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,535 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:38,535 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:38,536 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:38,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:38,536 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:38,536 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:38,536 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:38,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:38,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:38,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:38,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:38,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:38,538 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:38,538 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:38,538 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:38,538 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:38,538 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:38,539 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,539 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,539 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:38,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:38,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:38,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,539 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:38,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:38,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:38,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:38,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,540 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:38,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,542 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:38,542 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:38,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:38,542 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:38,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:38,542 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:38,542 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,542 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,542 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,542 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,542 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,542 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:38,542 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:38,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:38,542 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:38,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:38,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:38,551 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/83181b7025a24918b3869a25655eb522 2023-04-28 14:58:38,643 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:38,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:38,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:38,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:38,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:38,645 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:38,645 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:38,645 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:38,645 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:38,646 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,646 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,646 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:38,646 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:38,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:38,647 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,647 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:38,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:38,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,647 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:38,647 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:38,647 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:38,647 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:38,648 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:38,648 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:38,648 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:38,648 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:38,648 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:38,648 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:38,648 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:38,648 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:38,648 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:38,648 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:38,648 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,648 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:38,648 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:38,649 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,649 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:38,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:38,649 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,649 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:38,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:38,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:38,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:38,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:38,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7434f7d[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:38,650 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:38,650 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,651 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,651 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:38,651 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:38,651 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:38,651 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,651 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:38,652 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,652 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:38,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:38,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:38,652 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:38,652 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:38,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:38,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:38,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:38,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:38,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:38,653 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:38,653 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:38,653 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:38,653 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:38,653 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:38,654 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,654 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:38,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:38,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,654 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:38,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:38,655 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:38,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,656 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:38,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,657 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,657 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,657 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:38,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:38,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:38,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:38,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:38,657 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:38,657 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:38,657 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:38,657 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:38,657 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:38,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:38,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:38,657 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,658 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,658 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,658 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,757 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:38,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:38,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:38,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:38,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:38,760 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:38,760 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:38,760 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:38,760 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:38,761 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,761 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,761 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:38,761 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:38,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:38,762 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,762 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:38,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:38,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,762 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:38,762 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:38,762 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:38,763 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:38,763 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:38,763 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:38,763 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:38,763 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:38,763 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:38,763 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:38,763 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:38,763 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:38,763 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:38,763 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:38,763 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,763 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:38,763 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:38,764 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,764 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,764 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:38,765 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:38,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:38,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:38,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:38,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:38,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:38,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1399d3df[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:38,766 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:38,766 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,766 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,767 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,767 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:38,767 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:38,767 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:38,767 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:38,767 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:38,767 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,767 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:38,768 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:38,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:38,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:38,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:38,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:38,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:38,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:38,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:38,769 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:38,769 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:38,769 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:38,769 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:38,769 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:38,770 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,770 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:38,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:38,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,770 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:38,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:38,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:38,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:38,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,774 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:38,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,775 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:38,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,776 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:38,776 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:38,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:38,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:38,776 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,776 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,776 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,777 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,777 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,777 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:38,777 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:38,777 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:38,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:38,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:38,778 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:38,778 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:38,878 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:38,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:38,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:38,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:38,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:38,885 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:38,885 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:38,885 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:38,885 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:38,886 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,886 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,886 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,886 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:38,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:38,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:38,886 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,887 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:38,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:38,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,887 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:38,887 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:38,887 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:38,887 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:38,888 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:38,888 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:38,888 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:38,888 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:38,888 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:38,888 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:38,888 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:38,888 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:38,888 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:38,888 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:38,888 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,888 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:38,888 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:38,889 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,889 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:38,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:38,889 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,889 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:38,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:38,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:38,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:38,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:38,890 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:38,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7517cd3c[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:38,890 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,892 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,892 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:38,892 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:38,892 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:38,892 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,892 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:38,893 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,893 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:38,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:38,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:38,893 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:38,893 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:38,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:38,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:38,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:38,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:38,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:38,895 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:38,895 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:38,895 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:38,895 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:38,895 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:38,896 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,896 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,896 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:38,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:38,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:38,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:38,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:38,897 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:38,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:38,898 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:38,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:38,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,900 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,900 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:38,900 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:38,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:38,900 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:38,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:38,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:38,900 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:38,901 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:38,901 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:38,900 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:38,901 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:38,901 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:38,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:38,901 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:38,901 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:38,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:38,959 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/064f1cf2d5d24f8782a4640695f1b519 2023-04-28 14:58:38,963 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/3bb39b3d938f45bca45ed1c7c5c33b1b as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/3bb39b3d938f45bca45ed1c7c5c33b1b 2023-04-28 14:58:38,966 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/3bb39b3d938f45bca45ed1c7c5c33b1b, entries=250, sequenceid=192, filesize=38.9 K 2023-04-28 14:58:38,967 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/83181b7025a24918b3869a25655eb522 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/83181b7025a24918b3869a25655eb522 2023-04-28 14:58:38,970 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/83181b7025a24918b3869a25655eb522, entries=150, sequenceid=192, filesize=11.8 K 2023-04-28 14:58:38,971 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/064f1cf2d5d24f8782a4640695f1b519 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/064f1cf2d5d24f8782a4640695f1b519 2023-04-28 14:58:38,974 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/064f1cf2d5d24f8782a4640695f1b519, entries=150, sequenceid=192, filesize=11.8 K 2023-04-28 14:58:38,974 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~420.80 KB/430896, currentSize=33.54 KB/34350 for 1b6356a2d73db55249b3f0cb6bbd108a in 1647ms, sequenceid=192, compaction requested=true 2023-04-28 14:58:38,974 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:38,974 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:38,974 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 14:58:38,974 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 14:58:38,974 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:38,975 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:38,976 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 143302 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 14:58:38,976 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): 1b6356a2d73db55249b3f0cb6bbd108a/A is initiating minor compaction (all files) 2023-04-28 14:58:38,976 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1b6356a2d73db55249b3f0cb6bbd108a/A in TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:38,976 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/904d6b2d9cf44200a3f6a70cee5ec6e1, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/1e5211d6c9be47d2af90f35ae1d19606, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/ee9b85658f1943e7b84b049563af6afe, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/d45bf5f28f9f4c299e1d92de9c65c639, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/3bb39b3d938f45bca45ed1c7c5c33b1b] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp, totalSize=139.9 K 2023-04-28 14:58:38,976 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60018 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 14:58:38,976 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 1b6356a2d73db55249b3f0cb6bbd108a/B is initiating minor compaction (all files) 2023-04-28 14:58:38,976 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1b6356a2d73db55249b3f0cb6bbd108a/B in TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:38,976 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 904d6b2d9cf44200a3f6a70cee5ec6e1, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1682693910956 2023-04-28 14:58:38,976 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/7139dae4f3a847f6983d39f5795f45a7, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/2a9b9c1d30a641ae8df30685f492177d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/e947743dfcbf4590a7a29276fae44ea4, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/50e702134ae74dd6884303929eb10233, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/83181b7025a24918b3869a25655eb522] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp, totalSize=58.6 K 2023-04-28 14:58:38,976 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 1e5211d6c9be47d2af90f35ae1d19606, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1682693911594 2023-04-28 14:58:38,976 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 7139dae4f3a847f6983d39f5795f45a7, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1682693910956 2023-04-28 14:58:38,976 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting ee9b85658f1943e7b84b049563af6afe, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1682693912799 2023-04-28 14:58:38,977 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 2a9b9c1d30a641ae8df30685f492177d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1682693911594 2023-04-28 14:58:38,977 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting d45bf5f28f9f4c299e1d92de9c65c639, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1682693913934 2023-04-28 14:58:38,977 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 3bb39b3d938f45bca45ed1c7c5c33b1b, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1682693916187 2023-04-28 14:58:38,977 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting e947743dfcbf4590a7a29276fae44ea4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1682693912799 2023-04-28 14:58:38,977 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 50e702134ae74dd6884303929eb10233, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1682693913934 2023-04-28 14:58:38,978 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 83181b7025a24918b3869a25655eb522, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1682693916196 2023-04-28 14:58:38,992 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 1b6356a2d73db55249b3f0cb6bbd108a#B#compaction#310 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:38,993 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1b6356a2d73db55249b3f0cb6bbd108a#A#compaction#309 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:39,001 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:39,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:39,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:39,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:39,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:39,005 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:39,005 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:39,005 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:39,005 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:39,005 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:39,005 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:39,006 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:39,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:39,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:39,006 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:39,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:39,006 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:39,006 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:39,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:39,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:39,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:39,006 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:39,007 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:39,007 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:39,007 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:39,007 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:39,007 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:39,007 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:39,007 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:39,007 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:39,007 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:39,007 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1b6356a2d73db55249b3f0cb6bbd108a 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-28 14:58:39,007 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=A 2023-04-28 14:58:39,007 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:39,007 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=B 2023-04-28 14:58:39,007 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:39,007 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=C 2023-04-28 14:58:39,007 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:39,404 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/0babcce5a49e4211adb6c01503200b41 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/0babcce5a49e4211adb6c01503200b41 2023-04-28 14:58:39,407 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 1b6356a2d73db55249b3f0cb6bbd108a/B of 1b6356a2d73db55249b3f0cb6bbd108a into 0babcce5a49e4211adb6c01503200b41(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:39,407 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:39,407 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a., storeName=1b6356a2d73db55249b3f0cb6bbd108a/B, priority=11, startTime=1682693918974; duration=0sec 2023-04-28 14:58:39,407 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:39,407 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 14:58:39,415 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60018 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 14:58:39,415 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 1b6356a2d73db55249b3f0cb6bbd108a/C is initiating minor compaction (all files) 2023-04-28 14:58:39,415 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1b6356a2d73db55249b3f0cb6bbd108a/C in TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:39,415 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/17b70c903b714e83a2fdab80eaa83098, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/52fd3a4b770b4489b78a838d85796a05, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/cf3ba797893847148b398d7d16af6573, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/f660052b0d1045f7893cdfcd2d5ee1b4, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/064f1cf2d5d24f8782a4640695f1b519] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp, totalSize=58.6 K 2023-04-28 14:58:39,416 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 17b70c903b714e83a2fdab80eaa83098, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1682693910956 2023-04-28 14:58:39,416 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 52fd3a4b770b4489b78a838d85796a05, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1682693911594 2023-04-28 14:58:39,416 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting cf3ba797893847148b398d7d16af6573, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1682693912799 2023-04-28 14:58:39,417 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting f660052b0d1045f7893cdfcd2d5ee1b4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1682693913934 2023-04-28 14:58:39,417 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 064f1cf2d5d24f8782a4640695f1b519, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1682693916196 2023-04-28 14:58:39,423 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 1b6356a2d73db55249b3f0cb6bbd108a#C#compaction#312 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:39,430 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/f71e25b0a19744d893aaaf6a2ea37d2e as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/f71e25b0a19744d893aaaf6a2ea37d2e 2023-04-28 14:58:39,432 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304281eece70ae65644b9adea49454716aab7 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304281eece70ae65644b9adea49454716aab7 2023-04-28 14:58:39,434 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 1b6356a2d73db55249b3f0cb6bbd108a/C of 1b6356a2d73db55249b3f0cb6bbd108a into f71e25b0a19744d893aaaf6a2ea37d2e(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:39,434 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:39,434 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a., storeName=1b6356a2d73db55249b3f0cb6bbd108a/C, priority=11, startTime=1682693918975; duration=0sec 2023-04-28 14:58:39,434 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:39,437 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=200, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/9af92d3dfb304e17bc935f50388c0840 2023-04-28 14:58:39,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:39,488 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:39,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693919500, max timestamp=1682693919521Num uniques 859; 2023-04-28 14:58:39,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:58:39,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693919500, max timestamp=1682693919521Num uniques 859; 2023-04-28 14:58:39,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693919500, max timestamp=1682693919521Num uniques 859; 2023-04-28 14:58:39,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:58:39,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:58:39,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682693919560, max timestamp=1682693919578Num uniques 1018; 2023-04-28 14:58:39,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-28 14:58:39,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682693919560, max timestamp=1682693919578Num uniques 1018; 2023-04-28 14:58:39,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-28 14:58:39,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682693919560, max timestamp=1682693919578Num uniques 1018; 2023-04-28 14:58:39,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-28 14:58:39,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1682693919607, max timestamp=1682693919623Num uniques 1077; 2023-04-28 14:58:39,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-28 14:58:39,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1682693919607, max timestamp=1682693919623Num uniques 1077; 2023-04-28 14:58:39,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1682693919607, max timestamp=1682693919623Num uniques 1077; 2023-04-28 14:58:39,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-28 14:58:39,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-28 14:58:39,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:39,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693979692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:39,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:39,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37478 deadline: 1682693979695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:39,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:39,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693979800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:39,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:39,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37478 deadline: 1682693979805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:39,844 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/4ce3f94d2e0842c5a6142964a664ee2f 2023-04-28 14:58:40,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:40,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693980008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:40,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37478 deadline: 1682693980012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,216 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/d886632b0c194a4e8edc48db9c3b0da5 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/d886632b0c194a4e8edc48db9c3b0da5 2023-04-28 14:58:40,224 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 1b6356a2d73db55249b3f0cb6bbd108a/A of 1b6356a2d73db55249b3f0cb6bbd108a into d886632b0c194a4e8edc48db9c3b0da5(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-28 14:58:40,224 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:40,224 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a., storeName=1b6356a2d73db55249b3f0cb6bbd108a/A, priority=11, startTime=1682693918974; duration=1sec 2023-04-28 14:58:40,224 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:40,259 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/c21b06c7d9af40ada1ff3cd7f572c583 2023-04-28 14:58:40,262 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/9af92d3dfb304e17bc935f50388c0840 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/9af92d3dfb304e17bc935f50388c0840 2023-04-28 14:58:40,265 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/9af92d3dfb304e17bc935f50388c0840, entries=150, sequenceid=200, filesize=25.3 K 2023-04-28 14:58:40,266 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/4ce3f94d2e0842c5a6142964a664ee2f as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/4ce3f94d2e0842c5a6142964a664ee2f 2023-04-28 14:58:40,269 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/4ce3f94d2e0842c5a6142964a664ee2f, entries=150, sequenceid=200, filesize=11.8 K 2023-04-28 14:58:40,269 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/c21b06c7d9af40ada1ff3cd7f572c583 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/c21b06c7d9af40ada1ff3cd7f572c583 2023-04-28 14:58:40,273 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/c21b06c7d9af40ada1ff3cd7f572c583, entries=150, sequenceid=200, filesize=11.8 K 2023-04-28 14:58:40,274 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 1b6356a2d73db55249b3f0cb6bbd108a in 1267ms, sequenceid=200, compaction requested=false 2023-04-28 14:58:40,274 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:40,274 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:40,274 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:40,274 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:40,274 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,274 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:40,274 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:40,275 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,275 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:40,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:40,275 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,275 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:40,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:40,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:40,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:40,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:40,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36305f73[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:40,276 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:40,276 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,277 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,277 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:40,277 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:40,277 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:40,277 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,277 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:40,278 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:40,278 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:40,278 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:40,278 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:40,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:40,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:40,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:40,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:40,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:40,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:40,280 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:40,280 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:40,280 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:40,280 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:40,280 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:40,281 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,281 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,281 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:40,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:40,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:40,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:40,281 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:40,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:40,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:40,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,283 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:40,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,285 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,285 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,285 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,285 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,285 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,285 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,285 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:40,285 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:40,285 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:40,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:40,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:40,285 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:40,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:40,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:40,285 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:40,285 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:40,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:40,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:40,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:40,315 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1b6356a2d73db55249b3f0cb6bbd108a 3/3 column families, dataSize=154.31 KB heapSize=434.39 KB 2023-04-28 14:58:40,316 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=A 2023-04-28 14:58:40,316 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 14:58:40,316 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=B 2023-04-28 14:58:40,316 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 14:58:40,316 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=C 2023-04-28 14:58:40,316 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 14:58:40,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:40,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693980326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:40,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37478 deadline: 1682693980328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,385 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:40,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:40,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:40,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:40,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:40,388 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:40,388 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:40,388 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:40,388 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:40,388 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,388 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,389 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:40,389 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:40,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:40,389 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,389 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:40,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:40,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:40,390 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:40,390 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:40,390 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:40,390 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:40,390 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:40,394 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:40,394 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:40,394 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:40,394 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:40,394 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:40,394 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:40,394 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:40,394 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:40,394 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,394 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:40,394 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:40,395 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,395 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:40,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:40,396 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,396 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:40,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:40,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:40,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:40,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:40,397 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:40,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b15c66f[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:40,397 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,397 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,397 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:40,397 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:40,398 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,398 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:40,398 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:40,398 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:40,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:40,398 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:40,398 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:40,399 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:40,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:40,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:40,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:40,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:40,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:40,400 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:40,400 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:40,400 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:40,400 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:40,400 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:40,401 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,401 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,401 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:40,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:40,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:40,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:40,402 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:40,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,403 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:40,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,404 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,404 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:40,404 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,404 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:40,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:40,404 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:40,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:40,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:40,404 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,404 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,404 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,404 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,405 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,405 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:40,405 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:40,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:40,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:40,433 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:40,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693980431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:40,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37478 deadline: 1682693980431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,505 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:40,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:40,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:40,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:40,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:40,507 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:40,507 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:40,507 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:40,507 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:40,507 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,507 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,508 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,508 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:40,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:40,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:40,508 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,508 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:40,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:40,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,509 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:40,509 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:40,509 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:40,509 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:40,509 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:40,509 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:40,509 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:40,509 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:40,509 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:40,510 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:40,510 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:40,510 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:40,510 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:40,510 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:40,510 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,510 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:40,510 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:40,510 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,510 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:40,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:40,511 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,511 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:40,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:40,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:40,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:40,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:40,512 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:40,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@764feff9[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:40,512 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,512 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,513 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:40,513 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:40,513 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:40,513 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,513 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:40,513 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:40,513 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:40,513 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,513 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:40,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:40,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:40,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:40,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:40,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:40,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:40,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:40,515 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:40,516 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:40,516 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:40,516 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:40,516 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:40,516 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,516 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,516 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:40,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:40,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:40,518 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:40,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,518 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:40,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,520 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,520 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,520 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,520 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,520 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:40,520 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:40,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:40,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:40,520 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:40,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:40,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:40,520 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:40,520 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:40,520 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:40,520 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,520 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:40,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:40,620 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:40,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:40,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:40,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:40,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:40,623 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:40,623 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:40,623 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:40,623 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:40,623 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,623 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,625 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,625 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:40,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:40,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:40,625 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,625 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:40,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:40,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,626 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:40,626 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:40,626 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:40,626 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:40,626 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:40,626 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:40,626 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:40,626 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:40,626 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:40,626 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:40,626 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:40,626 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:40,626 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:40,626 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:40,626 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,626 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:40,626 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:40,627 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,627 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:40,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:40,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:40,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:40,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,628 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,628 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:40,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:40,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:40,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d7ab221[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:40,629 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:40,629 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,629 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,629 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:40,629 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:40,629 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:40,629 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,630 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:40,630 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:40,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:40,630 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:40,631 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:40,631 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:40,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:40,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:40,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:40,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:40,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:40,632 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:40,632 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:40,632 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:40,632 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:40,632 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:40,633 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,633 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,633 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:40,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:40,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:40,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:40,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:40,634 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:40,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:40,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,636 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:40,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,637 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,637 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,637 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:40,637 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:40,637 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,637 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:40,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:40,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:40,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:40,637 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:40,637 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:40,637 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,637 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:40,637 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,637 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:40,637 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:40,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:40,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:40,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693980634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:40,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37478 deadline: 1682693980634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,728 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428770d9ccc6c7e4d0d9895af12d656737c to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428770d9ccc6c7e4d0d9895af12d656737c 2023-04-28 14:58:40,737 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:40,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:40,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:40,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:40,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:40,740 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:40,740 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:40,740 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:40,740 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:40,740 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,740 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,741 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,741 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,741 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:40,741 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:40,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:40,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:40,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:40,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,742 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:40,743 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:40,743 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:40,743 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:40,743 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:40,743 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:40,743 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:40,743 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:40,743 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:40,743 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:40,743 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:40,743 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:40,743 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:40,743 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:40,743 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,743 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:40,743 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:40,744 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,744 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:40,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:40,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:40,744 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,744 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:40,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:40,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:40,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:40,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@905b93a[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:40,745 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:40,745 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,746 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,746 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,746 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:40,746 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:40,746 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:40,746 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:40,747 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:40,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:40,747 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:40,747 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:40,747 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:40,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:40,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:40,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:40,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:40,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:40,748 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:40,748 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:40,748 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:40,749 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:40,749 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:40,749 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,749 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,749 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:40,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:40,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:40,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,750 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:40,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:40,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:40,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:40,751 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:40,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,752 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,752 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,752 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,752 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:40,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:40,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:40,752 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:40,752 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,752 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,752 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,752 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:40,752 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:40,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:40,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:40,752 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:40,752 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:40,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:40,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:40,852 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:40,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:40,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:40,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:40,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:40,855 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:40,855 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:40,855 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:40,855 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:40,856 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,856 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,856 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,856 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:40,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:40,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:40,857 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,857 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:40,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:40,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,857 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:40,858 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:40,858 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:40,858 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:40,858 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:40,858 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:40,858 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:40,858 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:40,858 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:40,858 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:40,858 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:40,858 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:40,858 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:40,858 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:40,858 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,858 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:40,858 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:40,859 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:40,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:40,859 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:40,859 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,859 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:40,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:40,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:40,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:40,860 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:40,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ca630cb[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:40,860 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,861 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,861 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:40,861 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,861 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:40,861 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:40,861 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:40,862 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,862 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:40,862 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:40,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:40,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:40,862 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:40,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:40,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:40,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:40,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:40,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:40,864 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:40,864 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:40,864 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:40,864 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:40,864 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:40,865 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:40,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:40,865 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,865 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:40,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:40,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:40,865 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:40,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:40,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,866 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:40,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,868 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,868 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,868 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,868 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,868 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,868 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:40,868 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:40,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:40,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:40,868 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,868 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:40,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:40,868 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:40,868 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:40,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:40,868 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:40,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:40,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:40,947 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:40,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37478 deadline: 1682693980945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:40,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693980945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,968 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:40,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:40,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:40,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:40,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:40,971 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:40,971 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:40,971 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:40,971 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:40,971 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,971 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,972 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:40,972 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:40,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:40,972 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,972 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:40,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:40,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,973 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:40,973 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:40,973 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:40,973 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:40,973 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:40,973 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:40,973 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:40,973 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:40,973 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:40,973 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:40,973 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:40,973 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:40,973 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:40,973 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:40,973 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,973 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:40,973 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:40,975 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,975 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:40,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:40,975 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,975 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:40,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:40,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:40,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:40,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:40,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59f2681[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:40,976 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:40,976 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,977 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:40,977 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:40,977 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:40,978 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,978 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:40,978 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,978 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:40,978 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:40,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:40,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:40,978 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:40,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:40,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:40,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:40,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:40,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:40,980 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:40,980 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:40,980 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:40,980 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:40,980 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:40,981 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,981 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:40,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:40,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,981 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:40,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:40,981 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:40,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:40,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:40,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:40,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,983 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:40,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,986 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:40,986 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,986 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:40,986 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:40,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:40,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:40,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:40,986 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:40,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:40,986 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:40,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:40,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:40,986 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:40,986 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:40,987 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,987 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:40,987 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:40,987 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,087 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:41,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:41,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:41,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:41,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:41,089 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:41,089 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:41,089 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:41,089 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:41,090 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,090 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,090 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:41,090 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:41,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:41,091 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,091 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:41,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:41,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,091 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:41,091 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:41,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:41,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:41,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:41,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:41,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:41,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:41,092 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:41,092 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:41,092 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:41,092 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:41,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:41,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:41,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:41,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:41,093 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,093 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:41,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:41,093 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,093 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:41,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:41,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:41,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:41,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:41,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d9c77a9[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:41,094 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:41,095 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,095 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,095 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:41,095 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:41,095 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:41,095 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,095 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:41,096 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:41,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:41,096 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:41,096 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:41,096 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:41,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:41,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:41,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:41,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:41,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:41,098 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:41,098 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:41,098 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:41,098 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:41,098 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:41,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:41,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:41,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:41,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:41,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:41,100 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:41,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:41,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,100 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:41,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,102 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,102 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:41,102 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,102 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:41,102 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:41,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:41,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:41,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:41,102 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:41,102 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:41,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:41,102 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,102 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:41,102 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:41,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:41,102 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,102 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,132 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=275, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/fe328da62af74718957712487a1c53ab 2023-04-28 14:58:41,202 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:41,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:41,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:41,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:41,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:41,205 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:41,205 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:41,205 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:41,205 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:41,205 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,205 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,206 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,206 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:41,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:41,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:41,207 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,207 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:41,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:41,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,207 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:41,207 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:41,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:41,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:41,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:41,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:41,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:41,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:41,208 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:41,208 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:41,208 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:41,208 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:41,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:41,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:41,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:41,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:41,209 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,209 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,209 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:41,210 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:41,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:41,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:41,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:41,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:41,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:41,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49f4cf49[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:41,211 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:41,211 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,212 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,212 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:41,212 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,212 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:41,212 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:41,212 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:41,214 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:41,214 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,214 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:41,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:41,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:41,214 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:41,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:41,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:41,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:41,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:41,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:41,217 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:41,217 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:41,217 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:41,217 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:41,217 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:41,218 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:41,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:41,218 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,218 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:41,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:41,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:41,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:41,219 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:41,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,220 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:41,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,222 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:41,222 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:41,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:41,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:41,222 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,222 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,222 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,222 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,222 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:41,222 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,222 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,222 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:41,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:41,223 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:41,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:41,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:41,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:41,223 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:41,323 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:41,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:41,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:41,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:41,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:41,326 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:41,326 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:41,326 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:41,326 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:41,326 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,326 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,327 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,327 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:41,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:41,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:41,327 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,327 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:41,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:41,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,328 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:41,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:41,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:41,328 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:41,328 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:41,328 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:41,329 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:41,329 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:41,329 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:41,329 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:41,329 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:41,329 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:41,329 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:41,330 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:41,330 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,330 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:41,330 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:41,331 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,331 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:41,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:41,331 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,331 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:41,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:41,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:41,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:41,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:41,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74e98ed4[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:41,332 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:41,333 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,333 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,333 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:41,333 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,333 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:41,333 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:41,333 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:41,335 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,335 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:41,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:41,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:41,335 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:41,335 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:41,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:41,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:41,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:41,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:41,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:41,337 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:41,337 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:41,337 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:41,337 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:41,337 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:41,339 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:41,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:41,339 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,340 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:41,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:41,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:41,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:41,340 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:41,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,341 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:41,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,342 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,342 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,342 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,342 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:41,342 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:41,342 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:41,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:41,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:41,342 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,343 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,343 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,343 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:41,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:41,343 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:41,343 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:41,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:41,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:41,443 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:41,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:41,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:41,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:41,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:41,446 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:41,446 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:41,446 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:41,446 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:41,446 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,446 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,447 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,447 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:41,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:41,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:41,447 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,447 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:41,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:41,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,448 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:41,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:41,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:41,448 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:41,448 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:41,448 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:41,448 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:41,448 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:41,448 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:41,448 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:41,449 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:41,449 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:41,449 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:41,449 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:41,449 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,449 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:41,449 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:41,450 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,450 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:41,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:41,450 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,450 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:41,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:41,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:41,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:41,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:41,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8be05e6[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:41,451 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:41,451 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,452 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,453 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:41,453 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:41,453 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:41,453 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,453 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:41,454 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:41,454 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:41,454 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:41,454 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:41,454 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:41,454 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:41,454 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:41,454 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:41,454 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:41,455 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,455 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:41,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:41,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,455 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:41,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:41,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37478 deadline: 1682693981451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:41,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:41,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:41,455 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:41,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,457 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:41,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,458 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:41,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693981454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,458 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:41,458 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:41,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:41,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:41,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:41,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:41,459 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,459 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,459 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,459 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,459 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,459 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,459 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:41,459 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:41,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:41,459 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:41,459 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:41,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:41,542 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/d11afaf9a1fc4548b1ec0dd7a3ce858f 2023-04-28 14:58:41,559 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:41,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:41,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:41,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:41,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:41,562 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:41,562 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:41,562 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:41,562 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:41,562 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,562 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,563 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,563 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:41,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:41,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:41,563 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,563 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:41,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:41,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,564 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:41,564 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:41,564 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:41,564 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:41,564 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:41,564 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:41,564 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:41,564 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:41,566 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:41,566 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:41,566 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:41,566 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:41,566 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:41,566 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:41,566 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,566 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:41,566 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:41,567 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,567 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:41,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:41,567 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,567 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:41,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:41,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:41,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:41,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:41,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6910a67[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:41,568 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:41,568 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,569 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,569 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:41,569 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:41,569 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:41,569 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,569 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:41,569 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:41,570 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:41,569 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,570 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:41,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:41,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:41,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:41,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:41,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:41,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:41,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:41,572 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:41,572 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:41,572 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:41,572 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:41,572 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:41,573 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,573 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:41,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:41,573 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:41,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:41,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:41,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:41,574 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:41,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,575 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:41,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,576 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:41,576 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,576 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:41,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:41,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:41,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:41,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:41,576 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:41,576 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,576 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,576 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,576 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:41,576 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,577 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:41,577 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,577 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:41,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:41,677 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:41,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:41,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:41,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:41,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:41,679 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:41,679 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:41,679 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:41,679 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:41,680 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,680 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,680 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:41,680 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:41,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:41,681 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,681 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:41,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:41,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:41,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:41,682 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:41,682 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:41,682 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:41,682 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:41,682 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:41,682 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:41,682 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:41,682 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:41,682 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:41,682 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:41,682 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:41,682 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,682 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:41,682 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:41,683 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,683 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:41,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:41,683 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,683 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:41,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:41,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:41,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:41,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:41,684 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:41,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69f0137b[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:41,684 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,685 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,685 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:41,685 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:41,685 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:41,685 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,685 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:41,686 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:41,686 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:41,686 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,686 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:41,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:41,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:41,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:41,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:41,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:41,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:41,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:41,688 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:41,688 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:41,688 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:41,688 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:41,688 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:41,690 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,690 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:41,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:41,690 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:41,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:41,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:41,690 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:41,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:41,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,691 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:41,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,695 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,695 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,695 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,695 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:41,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:41,695 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:41,695 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,695 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:41,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:41,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:41,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:41,695 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,695 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,695 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:41,695 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:41,695 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:41,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:41,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:41,795 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:41,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:41,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:41,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:41,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:41,798 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:41,798 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:41,798 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:41,798 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:41,799 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,799 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,799 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,799 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:41,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:41,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:41,799 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,799 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:41,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:41,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,800 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:41,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:41,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:41,800 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:41,800 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:41,800 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:41,800 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:41,800 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:41,801 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:41,800 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:41,801 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:41,801 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:41,801 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:41,801 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:41,801 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,801 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:41,801 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:41,801 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,801 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:41,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:41,802 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,802 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:41,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:41,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:41,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:41,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:41,803 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:41,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5157afbf[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:41,803 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,803 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,804 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:41,804 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:41,804 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:41,804 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,804 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:41,804 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,804 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:41,804 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:41,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:41,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:41,804 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:41,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:41,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:41,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:41,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:41,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:41,806 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:41,806 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:41,806 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:41,806 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:41,806 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:41,807 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,807 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:41,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:41,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,807 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:41,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:41,808 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:41,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:41,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:41,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,809 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:41,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,810 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,810 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,810 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:41,810 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:41,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:41,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:41,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:41,810 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:41,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:41,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:41,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:41,810 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:41,811 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,811 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,811 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,811 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,811 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:41,811 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:41,911 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:41,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:41,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:41,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:41,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:41,914 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:41,914 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:41,914 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:41,914 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:41,914 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,914 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,915 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,915 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:41,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:41,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:41,915 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,915 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:41,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,915 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:41,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,915 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:41,916 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:41,916 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:41,916 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:41,916 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:41,916 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:41,916 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:41,916 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:41,916 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:41,916 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:41,916 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:41,916 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:41,916 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:41,916 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:41,916 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,916 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:41,916 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:41,917 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,917 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:41,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:41,917 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,917 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:41,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:41,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:41,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:41,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:41,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29be0146[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:41,918 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:41,918 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,919 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,919 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:41,919 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:41,919 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:41,919 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,919 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:41,920 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:41,920 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:41,920 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,920 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:41,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:41,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:41,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:41,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:41,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:41,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:41,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:41,923 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:41,923 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:41,923 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:41,923 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:41,923 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:41,925 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,925 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,925 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:41,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:41,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:41,925 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:41,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:41,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:41,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:41,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,926 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:41,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:41,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,928 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:41,928 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:41,928 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,928 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,928 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:41,928 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:41,928 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,928 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:41,928 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:41,928 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:41,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:41,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:41,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:41,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:41,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:41,928 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:41,928 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:41,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:41,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:41,950 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/65829dd1d4d24cddabccacc16243210d 2023-04-28 14:58:41,953 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/fe328da62af74718957712487a1c53ab as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/fe328da62af74718957712487a1c53ab 2023-04-28 14:58:41,956 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/fe328da62af74718957712487a1c53ab, entries=200, sequenceid=275, filesize=32.3 K 2023-04-28 14:58:41,957 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/d11afaf9a1fc4548b1ec0dd7a3ce858f as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/d11afaf9a1fc4548b1ec0dd7a3ce858f 2023-04-28 14:58:41,960 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/d11afaf9a1fc4548b1ec0dd7a3ce858f, entries=150, sequenceid=275, filesize=11.9 K 2023-04-28 14:58:41,961 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/65829dd1d4d24cddabccacc16243210d as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/65829dd1d4d24cddabccacc16243210d 2023-04-28 14:58:41,964 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/65829dd1d4d24cddabccacc16243210d, entries=150, sequenceid=275, filesize=11.9 K 2023-04-28 14:58:41,965 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~434.34 KB/444768, currentSize=26.84 KB/27480 for 1b6356a2d73db55249b3f0cb6bbd108a in 1650ms, sequenceid=275, compaction requested=true 2023-04-28 14:58:41,965 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:41,965 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:41,965 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:41,965 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:58:41,965 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:41,965 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:58:41,966 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36566 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:58:41,966 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85186 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:58:41,966 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 1b6356a2d73db55249b3f0cb6bbd108a/B is initiating minor compaction (all files) 2023-04-28 14:58:41,966 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): 1b6356a2d73db55249b3f0cb6bbd108a/A is initiating minor compaction (all files) 2023-04-28 14:58:41,966 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1b6356a2d73db55249b3f0cb6bbd108a/B in TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:41,966 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1b6356a2d73db55249b3f0cb6bbd108a/A in TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:41,966 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/0babcce5a49e4211adb6c01503200b41, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/4ce3f94d2e0842c5a6142964a664ee2f, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/d11afaf9a1fc4548b1ec0dd7a3ce858f] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp, totalSize=35.7 K 2023-04-28 14:58:41,966 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/d886632b0c194a4e8edc48db9c3b0da5, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/9af92d3dfb304e17bc935f50388c0840, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/fe328da62af74718957712487a1c53ab] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp, totalSize=83.2 K 2023-04-28 14:58:41,967 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 0babcce5a49e4211adb6c01503200b41, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1682693916196 2023-04-28 14:58:41,967 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting d886632b0c194a4e8edc48db9c3b0da5, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1682693916196 2023-04-28 14:58:41,967 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 4ce3f94d2e0842c5a6142964a664ee2f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1682693917335 2023-04-28 14:58:41,967 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 9af92d3dfb304e17bc935f50388c0840, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1682693917335 2023-04-28 14:58:41,967 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting d11afaf9a1fc4548b1ec0dd7a3ce858f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1682693919678 2023-04-28 14:58:41,967 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting fe328da62af74718957712487a1c53ab, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1682693919678 2023-04-28 14:58:41,974 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 1b6356a2d73db55249b3f0cb6bbd108a#B#compaction#318 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:41,976 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1b6356a2d73db55249b3f0cb6bbd108a#A#compaction#319 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:41,993 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/daefd9e491814509abb81e77e3fa4b9f as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/daefd9e491814509abb81e77e3fa4b9f 2023-04-28 14:58:41,997 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1b6356a2d73db55249b3f0cb6bbd108a/B of 1b6356a2d73db55249b3f0cb6bbd108a into daefd9e491814509abb81e77e3fa4b9f(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:41,997 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:41,997 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a., storeName=1b6356a2d73db55249b3f0cb6bbd108a/B, priority=13, startTime=1682693921965; duration=0sec 2023-04-28 14:58:41,997 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:41,997 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:58:41,998 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36566 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:58:41,998 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 1b6356a2d73db55249b3f0cb6bbd108a/C is initiating minor compaction (all files) 2023-04-28 14:58:41,998 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1b6356a2d73db55249b3f0cb6bbd108a/C in TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:41,998 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/f71e25b0a19744d893aaaf6a2ea37d2e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/c21b06c7d9af40ada1ff3cd7f572c583, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/65829dd1d4d24cddabccacc16243210d] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp, totalSize=35.7 K 2023-04-28 14:58:41,998 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting f71e25b0a19744d893aaaf6a2ea37d2e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1682693916196 2023-04-28 14:58:41,999 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting c21b06c7d9af40ada1ff3cd7f572c583, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1682693917335 2023-04-28 14:58:41,999 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 65829dd1d4d24cddabccacc16243210d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1682693919678 2023-04-28 14:58:42,014 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 1b6356a2d73db55249b3f0cb6bbd108a#C#compaction#320 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:42,026 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/f7b9295342274723bd36e96bb9ca0f75 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/f7b9295342274723bd36e96bb9ca0f75 2023-04-28 14:58:42,028 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:42,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:42,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:42,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:42,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:42,032 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:42,032 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:42,032 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:42,032 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:42,032 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:42,032 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:42,033 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:42,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:42,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:42,033 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1b6356a2d73db55249b3f0cb6bbd108a/C of 1b6356a2d73db55249b3f0cb6bbd108a into f7b9295342274723bd36e96bb9ca0f75(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:42,033 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:42,033 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:42,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:42,033 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a., storeName=1b6356a2d73db55249b3f0cb6bbd108a/C, priority=13, startTime=1682693921965; duration=0sec 2023-04-28 14:58:42,033 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:42,033 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:42,033 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:42,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:42,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:42,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:42,034 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:42,034 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:42,034 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:42,034 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:42,034 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:42,034 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:42,034 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:42,034 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:42,034 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:42,034 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:42,034 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1b6356a2d73db55249b3f0cb6bbd108a 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-28 14:58:42,035 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=A 2023-04-28 14:58:42,035 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:42,035 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=B 2023-04-28 14:58:42,035 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:42,035 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=C 2023-04-28 14:58:42,035 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:42,409 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/fb08c9f4f01d4d0a93e8c56cdd15eb52 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/fb08c9f4f01d4d0a93e8c56cdd15eb52 2023-04-28 14:58:42,412 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1b6356a2d73db55249b3f0cb6bbd108a/A of 1b6356a2d73db55249b3f0cb6bbd108a into fb08c9f4f01d4d0a93e8c56cdd15eb52(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:42,412 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:42,413 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a., storeName=1b6356a2d73db55249b3f0cb6bbd108a/A, priority=13, startTime=1682693921965; duration=0sec 2023-04-28 14:58:42,413 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:42,445 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042808b69a7b574f456ea1f20e25f775867f to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042808b69a7b574f456ea1f20e25f775867f 2023-04-28 14:58:42,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:42,464 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:42,494 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693922482, max timestamp=1682693922493Num uniques 859; 2023-04-28 14:58:42,494 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:58:42,495 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693922482, max timestamp=1682693922493Num uniques 859; 2023-04-28 14:58:42,495 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:58:42,495 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693922482, max timestamp=1682693922493Num uniques 859; 2023-04-28 14:58:42,495 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:58:42,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682693922531, max timestamp=1682693922557Num uniques 1068; 2023-04-28 14:58:42,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-28 14:58:42,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682693922531, max timestamp=1682693922557Num uniques 1068; 2023-04-28 14:58:42,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682693922531, max timestamp=1682693922557Num uniques 1068; 2023-04-28 14:58:42,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-28 14:58:42,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-28 14:58:42,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682693922593, max timestamp=1682693922613Num uniques 1027; 2023-04-28 14:58:42,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682693922593, max timestamp=1682693922613Num uniques 1027; 2023-04-28 14:58:42,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682693922593, max timestamp=1682693922613Num uniques 1027; 2023-04-28 14:58:42,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-28 14:58:42,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-28 14:58:42,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-28 14:58:42,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682693922649, max timestamp=1682693922661Num uniques 1086; 2023-04-28 14:58:42,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-28 14:58:42,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682693922649, max timestamp=1682693922661Num uniques 1086; 2023-04-28 14:58:42,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682693922649, max timestamp=1682693922661Num uniques 1086; 2023-04-28 14:58:42,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-28 14:58:42,664 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-28 14:58:42,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682693922694, max timestamp=1682693922713Num uniques 1095; 2023-04-28 14:58:42,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-28 14:58:42,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682693922694, max timestamp=1682693922713Num uniques 1095; 2023-04-28 14:58:42,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-28 14:58:42,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682693922694, max timestamp=1682693922713Num uniques 1095; 2023-04-28 14:58:42,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-28 14:58:42,772 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682693922750, max timestamp=1682693922769Num uniques 1054; 2023-04-28 14:58:42,772 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682693922750, max timestamp=1682693922769Num uniques 1054; 2023-04-28 14:58:42,772 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682693922750, max timestamp=1682693922769Num uniques 1054; 2023-04-28 14:58:42,772 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-28 14:58:42,772 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-28 14:58:42,772 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-28 14:58:42,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682693922804, max timestamp=1682693922820Num uniques 1063; 2023-04-28 14:58:42,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-28 14:58:42,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682693922804, max timestamp=1682693922820Num uniques 1063; 2023-04-28 14:58:42,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682693922804, max timestamp=1682693922820Num uniques 1063; 2023-04-28 14:58:42,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-28 14:58:42,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-28 14:58:42,849 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=284, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/fac70c9af9624b5a9934b7aeb3e9b5f6 2023-04-28 14:58:42,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682693922849, max timestamp=1682693922873Num uniques 1072; 2023-04-28 14:58:42,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682693922849, max timestamp=1682693922873Num uniques 1072; 2023-04-28 14:58:42,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-28 14:58:42,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-28 14:58:42,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682693922849, max timestamp=1682693922873Num uniques 1072; 2023-04-28 14:58:42,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-28 14:58:42,914 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682693922898, max timestamp=1682693922912Num uniques 1031; 2023-04-28 14:58:42,914 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-28 14:58:42,914 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682693922898, max timestamp=1682693922912Num uniques 1031; 2023-04-28 14:58:42,914 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-28 14:58:42,915 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682693922898, max timestamp=1682693922912Num uniques 1031; 2023-04-28 14:58:42,915 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-28 14:58:42,972 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1682693922931, max timestamp=1682693922970Num uniques 1090; 2023-04-28 14:58:42,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-28 14:58:42,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1682693922931, max timestamp=1682693922970Num uniques 1090; 2023-04-28 14:58:42,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-28 14:58:42,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1682693922931, max timestamp=1682693922970Num uniques 1090; 2023-04-28 14:58:42,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-28 14:58:43,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1682693923011, max timestamp=1682693923031Num uniques 1099; 2023-04-28 14:58:43,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-28 14:58:43,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1682693923011, max timestamp=1682693923031Num uniques 1099; 2023-04-28 14:58:43,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-28 14:58:43,034 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1682693923011, max timestamp=1682693923031Num uniques 1099; 2023-04-28 14:58:43,034 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-28 14:58:43,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682693923050, max timestamp=1682693923082Num uniques 1058; 2023-04-28 14:58:43,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-28 14:58:43,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682693923050, max timestamp=1682693923082Num uniques 1058; 2023-04-28 14:58:43,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-28 14:58:43,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682693923050, max timestamp=1682693923082Num uniques 1058; 2023-04-28 14:58:43,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-28 14:58:43,152 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1682693923118, max timestamp=1682693923150Num uniques 1067; 2023-04-28 14:58:43,152 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-28 14:58:43,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1682693923118, max timestamp=1682693923150Num uniques 1067; 2023-04-28 14:58:43,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-28 14:58:43,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1682693923118, max timestamp=1682693923150Num uniques 1067; 2023-04-28 14:58:43,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-28 14:58:43,198 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:43,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693983193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682693923170, max timestamp=1682693923197Num uniques 1126; 2023-04-28 14:58:43,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=8056, suffix heap size=145072,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=176 2023-04-28 14:58:43,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682693923170, max timestamp=1682693923197Num uniques 1126; 2023-04-28 14:58:43,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=8056, suffix heap size=145072,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=176 2023-04-28 14:58:43,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682693923170, max timestamp=1682693923197Num uniques 1126; 2023-04-28 14:58:43,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=8056, suffix heap size=145072,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=176 2023-04-28 14:58:43,257 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/3f60af3c093c464187e51ee03191dcde 2023-04-28 14:58:43,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1682693923245, max timestamp=1682693923295Num uniques 1035; 2023-04-28 14:58:43,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-28 14:58:43,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1682693923245, max timestamp=1682693923295Num uniques 1035; 2023-04-28 14:58:43,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1682693923245, max timestamp=1682693923295Num uniques 1035; 2023-04-28 14:58:43,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-28 14:58:43,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-28 14:58:43,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1682693923331, max timestamp=1682693923347Num uniques 1044; 2023-04-28 14:58:43,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-28 14:58:43,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1682693923331, max timestamp=1682693923347Num uniques 1044; 2023-04-28 14:58:43,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-28 14:58:43,350 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1682693923331, max timestamp=1682693923347Num uniques 1044; 2023-04-28 14:58:43,350 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-28 14:58:43,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1682693923388, max timestamp=1682693923403Num uniques 1053; 2023-04-28 14:58:43,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=11585, suffix heap size=137480,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=253 2023-04-28 14:58:43,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1682693923388, max timestamp=1682693923403Num uniques 1053; 2023-04-28 14:58:43,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=11585, suffix heap size=137480,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=253 2023-04-28 14:58:43,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1682693923388, max timestamp=1682693923403Num uniques 1053; 2023-04-28 14:58:43,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=11585, suffix heap size=137480,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=253 2023-04-28 14:58:43,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1682693923421, max timestamp=1682693923447Num uniques 1112; 2023-04-28 14:58:43,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1682693923421, max timestamp=1682693923447Num uniques 1112; 2023-04-28 14:58:43,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=50922, new segment data size=9702, suffix heap size=143616,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1112, new segment cells count=212 2023-04-28 14:58:43,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=50922, new segment data size=9702, suffix heap size=143616,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1112, new segment cells count=212 2023-04-28 14:58:43,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1682693923421, max timestamp=1682693923447Num uniques 1112; 2023-04-28 14:58:43,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=50922, new segment data size=9702, suffix heap size=143616,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1112, new segment cells count=212 2023-04-28 14:58:43,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1682693923474, max timestamp=1682693923496Num uniques 1071; 2023-04-28 14:58:43,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=7826, suffix heap size=139352,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=171 2023-04-28 14:58:43,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1682693923474, max timestamp=1682693923496Num uniques 1071; 2023-04-28 14:58:43,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1682693923474, max timestamp=1682693923496Num uniques 1071; 2023-04-28 14:58:43,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=7826, suffix heap size=139352,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=171 2023-04-28 14:58:43,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=7826, suffix heap size=139352,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=171 2023-04-28 14:58:43,548 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1682693923531, max timestamp=1682693923547Num uniques 1030; 2023-04-28 14:58:43,548 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=8240, suffix heap size=135088,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=180 2023-04-28 14:58:43,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1682693923531, max timestamp=1682693923547Num uniques 1030; 2023-04-28 14:58:43,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=8240, suffix heap size=135088,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=180 2023-04-28 14:58:43,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1682693923531, max timestamp=1682693923547Num uniques 1030; 2023-04-28 14:58:43,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=8240, suffix heap size=135088,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=180 2023-04-28 14:58:43,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1682693923574, max timestamp=1682693923582Num uniques 1039; 2023-04-28 14:58:43,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=10944, suffix heap size=136024,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=239 2023-04-28 14:58:43,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1682693923574, max timestamp=1682693923582Num uniques 1039; 2023-04-28 14:58:43,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=10944, suffix heap size=136024,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=239 2023-04-28 14:58:43,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1682693923574, max timestamp=1682693923582Num uniques 1039; 2023-04-28 14:58:43,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=10944, suffix heap size=136024,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=239 2023-04-28 14:58:43,631 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9068, totalHeapSize=20832, min timestamp=1682693923621, max timestamp=1682693923629Num uniques 1098; 2023-04-28 14:58:43,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=9068, suffix heap size=142160,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=198 2023-04-28 14:58:43,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9068, totalHeapSize=20832, min timestamp=1682693923621, max timestamp=1682693923629Num uniques 1098; 2023-04-28 14:58:43,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=9068, suffix heap size=142160,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=198 2023-04-28 14:58:43,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9068, totalHeapSize=20832, min timestamp=1682693923621, max timestamp=1682693923629Num uniques 1098; 2023-04-28 14:58:43,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=9068, suffix heap size=142160,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=198 2023-04-28 14:58:43,673 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/f3d5f8106ef941a8b561125e7f70d0c9 2023-04-28 14:58:43,677 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/fac70c9af9624b5a9934b7aeb3e9b5f6 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/fac70c9af9624b5a9934b7aeb3e9b5f6 2023-04-28 14:58:43,680 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/fac70c9af9624b5a9934b7aeb3e9b5f6, entries=150, sequenceid=284, filesize=25.4 K 2023-04-28 14:58:43,681 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/3f60af3c093c464187e51ee03191dcde as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/3f60af3c093c464187e51ee03191dcde 2023-04-28 14:58:43,682 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1682693923666, max timestamp=1682693923679Num uniques 1057; 2023-04-28 14:58:43,682 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=9475, suffix heap size=137896,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=207 2023-04-28 14:58:43,682 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1682693923666, max timestamp=1682693923679Num uniques 1057; 2023-04-28 14:58:43,682 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=9475, suffix heap size=137896,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=207 2023-04-28 14:58:43,682 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1682693923666, max timestamp=1682693923679Num uniques 1057; 2023-04-28 14:58:43,682 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=9475, suffix heap size=137896,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=207 2023-04-28 14:58:43,684 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/3f60af3c093c464187e51ee03191dcde, entries=150, sequenceid=284, filesize=11.9 K 2023-04-28 14:58:43,685 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/f3d5f8106ef941a8b561125e7f70d0c9 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/f3d5f8106ef941a8b561125e7f70d0c9 2023-04-28 14:58:43,688 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/f3d5f8106ef941a8b561125e7f70d0c9, entries=150, sequenceid=284, filesize=11.9 K 2023-04-28 14:58:43,688 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=33.54 KB/34350 for 1b6356a2d73db55249b3f0cb6bbd108a in 1654ms, sequenceid=284, compaction requested=false 2023-04-28 14:58:43,688 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:43,688 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:43,689 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:43,689 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:43,689 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,689 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:43,689 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:43,689 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:43,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:43,691 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:43,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:43,691 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:43,691 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:43,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:43,692 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1b6356a2d73db55249b3f0cb6bbd108a 3/3 column families, dataSize=53.67 KB heapSize=141.42 KB 2023-04-28 14:58:43,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:43,692 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=A 2023-04-28 14:58:43,692 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:43,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:43,692 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=B 2023-04-28 14:58:43,693 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:43,693 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=C 2023-04-28 14:58:43,693 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:43,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:43,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:43,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2772f424[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:43,694 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:43,694 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:43,695 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:43,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:43,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:43,695 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:43,695 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:43,695 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:43,695 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,695 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:43,701 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,701 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:43,701 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:43,701 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:43,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:43,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:43,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:43,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:43,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:43,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:43,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:43,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:43,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:43,704 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:43,704 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:43,704 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:43,704 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:43,704 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:43,705 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:43,705 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:43,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:43,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:43,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:43,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:43,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:43,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:43,705 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:43,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:43,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:43,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:43,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:43,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:43,706 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:43,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:43,708 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:43,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,712 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428ea8d17ea802e4885b8875cc686a5567f to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ea8d17ea802e4885b8875cc686a5567f 2023-04-28 14:58:43,712 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,712 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:43,712 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:43,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:43,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:43,712 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:43,712 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:43,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:43,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:43,712 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:43,712 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:43,712 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,713 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:43,713 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:43,713 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:43,713 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:43,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:43,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:43,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682693923728, max timestamp=1682693923763Num uniques 859; 2023-04-28 14:58:43,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682693923728, max timestamp=1682693923763Num uniques 859; 2023-04-28 14:58:43,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 14:58:43,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 14:58:43,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682693923728, max timestamp=1682693923763Num uniques 859; 2023-04-28 14:58:43,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 14:58:43,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:43,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 399 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693983793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,801 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:43,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 408 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37478 deadline: 1682693983799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,813 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:43,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:43,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:43,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:43,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:43,815 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:43,815 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:43,815 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:43,815 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:43,815 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:43,815 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:43,816 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,816 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:43,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:43,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:43,816 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,816 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:43,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:43,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:43,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:43,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:43,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:43,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:43,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:43,817 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:43,817 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:43,817 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:43,817 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:43,817 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:43,817 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:43,817 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:43,817 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:43,817 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:43,817 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:43,817 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:43,817 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,818 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:43,818 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:43,819 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,819 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:43,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:43,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:43,819 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:43,819 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:43,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:43,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:43,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:43,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:43,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:43,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ef0507a[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:43,822 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:43,822 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:43,822 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:43,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:43,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:43,822 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:43,822 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,822 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:43,822 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:43,822 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:43,823 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,823 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:43,823 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:43,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:43,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:43,823 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:43,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:43,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:43,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:43,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:43,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:43,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:43,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:43,825 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:43,825 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:43,825 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:43,825 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:43,825 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:43,826 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:43,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:43,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:43,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:43,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:43,826 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:43,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:43,827 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:43,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:43,827 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:43,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:43,827 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:43,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:43,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:43,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:43,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:43,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:43,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,829 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,829 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:43,829 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:43,829 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,829 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:43,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:43,829 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:43,829 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:43,829 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:43,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:43,829 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:43,829 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:43,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:43,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:43,829 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:43,829 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:43,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:43,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:43,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:43,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:43,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 401 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693983900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,912 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:43,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 410 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37478 deadline: 1682693983901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,930 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:43,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:43,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:43,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:43,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:43,934 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:43,934 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:43,934 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:43,934 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:43,935 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:43,936 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:43,936 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:43,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:43,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:43,936 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:43,937 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:43,937 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:43,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:43,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:43,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:43,938 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:43,938 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:43,938 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:43,938 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:43,938 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:43,938 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:43,938 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:43,938 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:43,938 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:43,938 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:43,938 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:43,938 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:43,938 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:43,938 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,938 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:43,938 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:43,940 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:43,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:43,940 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:43,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:43,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:43,941 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:43,941 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:43,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:43,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:43,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:43,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2236344c[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:43,944 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:43,944 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:43,945 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:43,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:43,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:43,945 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:43,945 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:43,945 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:43,946 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,946 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:43,946 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:43,947 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:43,947 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:43,947 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:43,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:43,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:43,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:43,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:43,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:43,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:43,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:43,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:43,961 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:43,961 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:43,961 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:43,961 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:43,961 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:43,962 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:43,962 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:43,962 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:43,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:43,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:43,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:43,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:43,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:43,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:43,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:43,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:43,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:43,965 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:43,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:43,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:43,966 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:43,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:43,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,969 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,969 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:43,969 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:43,969 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:43,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:43,969 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:43,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:43,969 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:43,969 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:43,969 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:43,969 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:43,969 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:43,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:43,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:43,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:43,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:43,970 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:43,970 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:44,070 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:44,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:44,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:44,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:44,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:44,072 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:44,073 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:44,073 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:44,073 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:44,073 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,073 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,073 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:44,073 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:44,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:44,074 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,074 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:44,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:44,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,074 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:44,075 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:44,075 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:44,075 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:44,075 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:44,075 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:44,075 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:44,075 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:44,075 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:44,075 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:44,075 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:44,075 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:44,075 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:44,075 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,075 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:44,075 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:44,076 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,076 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:44,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:44,076 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,076 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:44,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:44,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:44,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:44,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:44,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b5bb57d[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:44,077 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:44,077 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,077 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,078 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:44,078 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:44,078 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:44,078 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,078 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:44,078 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:44,078 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,078 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:44,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:44,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:44,078 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:44,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:44,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:44,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:44,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:44,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:44,080 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:44,080 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:44,080 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:44,080 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:44,080 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:44,080 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,080 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:44,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:44,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,081 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:44,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:44,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:44,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:44,081 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:44,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,082 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:44,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,083 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:44,083 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:44,083 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:44,084 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:44,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:44,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:44,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:44,084 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,084 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,084 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,084 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:44,084 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:44,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:44,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:44,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:44,084 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,084 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,084 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,111 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:44,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 403 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693984107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:44,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 412 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37478 deadline: 1682693984114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,129 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=688, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/9297d81e27c6462bb42b605912f5efe7 2023-04-28 14:58:44,184 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:44,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:44,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:44,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:44,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:44,187 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:44,187 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:44,187 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:44,187 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:44,187 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,187 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,188 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:44,188 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:44,188 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,188 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:44,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:44,189 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:44,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:44,189 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:44,189 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:44,189 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:44,189 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:44,189 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:44,189 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:44,189 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:44,189 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:44,189 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:44,189 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:44,189 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:44,190 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,190 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:44,190 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:44,190 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,190 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:44,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:44,191 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,191 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:44,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:44,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:44,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:44,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:44,192 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:44,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b38ade6[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:44,192 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,192 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,192 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:44,192 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:44,192 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,193 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:44,193 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:44,193 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,193 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:44,193 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:44,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:44,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:44,193 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:44,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:44,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:44,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:44,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:44,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:44,195 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:44,195 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:44,195 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:44,195 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:44,195 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:44,196 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,196 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,196 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:44,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:44,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:44,197 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,198 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:44,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,200 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,200 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,200 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:44,200 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:44,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:44,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:44,200 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:44,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:44,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:44,200 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:44,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:44,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:44,200 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,200 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,200 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,200 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:44,200 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,200 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:44,301 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:44,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:44,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:44,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:44,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:44,303 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:44,303 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:44,303 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:44,303 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:44,304 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,304 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,304 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:44,304 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:44,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:44,305 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,305 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:44,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:44,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:44,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:44,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:44,306 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:44,306 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:44,306 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:44,306 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:44,306 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:44,306 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:44,306 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:44,306 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:44,306 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:44,306 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:44,306 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:44,306 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,306 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:44,306 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:44,307 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,307 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:44,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:44,307 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,307 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:44,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:44,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:44,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:44,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:44,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@707577b[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:44,308 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:44,308 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,309 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,309 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:44,309 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:44,309 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,309 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:44,309 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:44,310 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:44,310 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:44,310 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:44,310 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:44,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:44,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:44,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:44,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:44,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:44,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:44,314 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:44,314 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:44,314 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:44,314 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:44,314 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:44,315 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,315 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,315 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:44,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:44,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:44,316 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:44,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:44,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:44,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:44,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,317 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:44,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,318 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:44,318 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,318 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,318 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:44,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:44,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:44,318 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:44,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:44,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:44,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:44,318 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,318 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,318 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:44,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:44,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:44,318 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,318 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,318 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:44,319 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:44,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 405 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693984412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,419 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:44,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:44,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:44,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:44,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:44,421 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:44,421 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:44,421 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:44,421 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:44,422 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,422 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,422 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:44,422 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:44,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:44,423 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,423 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:44,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:44,423 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:44,423 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:44,423 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:44,423 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:44,423 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:44,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:44,430 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:44,430 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:44,430 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:44,430 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:44,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 414 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37478 deadline: 1682693984420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,429 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:44,430 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:44,430 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:44,430 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:44,431 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,431 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:44,431 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:44,432 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,432 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:44,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:44,432 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,432 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:44,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:44,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:44,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:44,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:44,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14009783[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:44,433 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:44,433 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,433 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,434 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:44,434 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:44,434 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,434 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:44,434 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:44,434 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,434 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:44,435 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:44,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:44,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:44,435 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:44,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:44,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:44,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:44,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:44,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:44,437 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:44,437 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:44,437 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:44,437 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:44,437 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:44,438 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:44,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:44,438 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,438 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:44,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:44,438 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:44,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:44,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:44,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,439 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:44,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,442 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:44,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:44,442 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:44,442 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:44,442 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:44,442 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:44,442 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:44,442 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:44,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:44,442 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,442 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,443 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,443 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,443 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:44,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:44,537 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=688 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/7bf68f3bd95e4c8982732197b7a03b21 2023-04-28 14:58:44,542 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:44,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:44,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:44,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:44,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:44,545 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:44,545 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:44,545 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:44,545 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:44,546 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,546 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,547 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:44,547 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:44,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:44,547 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,547 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:44,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:44,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,548 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:44,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:44,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:44,548 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:44,548 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:44,548 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:44,548 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:44,548 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:44,548 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:44,548 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:44,548 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:44,548 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:44,548 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:44,548 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:44,548 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,548 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:44,548 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:44,550 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,550 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:44,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:44,550 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,550 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:44,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:44,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:44,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:44,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:44,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32d62823[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:44,551 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:44,551 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,552 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,552 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:44,552 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:44,552 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:44,552 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,552 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:44,552 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:44,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:44,553 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:44,553 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:44,553 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:44,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:44,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:44,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:44,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:44,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:44,554 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:44,554 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:44,554 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:44,554 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:44,554 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:44,555 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,555 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,555 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:44,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:44,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:44,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:44,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:44,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:44,556 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:44,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,557 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:44,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,558 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,558 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,558 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,558 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:44,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:44,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:44,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:44,558 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:44,558 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,559 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,559 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,559 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,559 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:44,558 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:44,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:44,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:44,559 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:44,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:44,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:44,659 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:44,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:44,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:44,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:44,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:44,662 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:44,662 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:44,662 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:44,662 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:44,662 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,662 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,663 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:44,663 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:44,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:44,663 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,663 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:44,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:44,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,664 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:44,664 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:44,664 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:44,664 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:44,664 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:44,664 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:44,664 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:44,664 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:44,664 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:44,664 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:44,664 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:44,664 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:44,664 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:44,664 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:44,664 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,665 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:44,665 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:44,665 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,665 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:44,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:44,665 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,666 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:44,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:44,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:44,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:44,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:44,667 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:44,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c0ff402[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:44,667 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,667 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,667 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:44,667 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:44,667 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:44,668 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,668 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:44,668 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:44,668 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:44,668 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:44,668 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:44,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:44,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:44,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:44,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:44,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:44,670 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:44,670 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:44,670 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:44,670 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:44,670 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:44,671 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:44,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:44,671 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,671 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:44,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:44,672 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:44,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:44,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:44,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,672 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:44,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,674 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,674 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,674 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,674 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,674 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,674 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:44,674 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:44,674 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:44,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:44,674 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,674 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:44,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:44,674 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:44,674 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:44,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:44,774 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:44,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:44,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:44,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:44,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:44,777 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:44,777 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:44,777 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:44,777 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:44,777 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,777 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,778 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,778 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:44,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:44,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:44,778 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,778 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:44,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:44,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,779 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:44,779 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:44,779 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:44,779 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:44,779 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:44,779 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:44,779 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:44,779 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:44,779 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:44,779 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:44,779 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:44,779 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:44,780 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:44,780 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:44,780 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,780 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:44,780 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:44,780 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,780 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:44,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:44,781 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,781 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:44,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:44,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:44,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:44,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:44,782 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:44,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18ec077[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:44,782 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,782 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,782 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:44,782 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:44,782 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:44,782 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,782 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:44,783 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,783 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:44,783 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:44,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:44,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:44,783 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:44,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:44,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:44,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:44,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:44,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:44,785 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:44,785 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:44,785 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:44,785 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:44,785 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:44,786 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,786 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:44,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:44,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,786 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:44,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:44,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:44,786 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:44,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:44,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,787 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:44,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,788 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,788 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,789 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,789 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,789 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,789 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,789 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:44,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:44,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:44,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:44,789 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,789 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:44,789 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:44,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:44,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:44,789 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:44,789 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:44,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:44,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:44,889 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:44,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:44,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:44,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:44,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:44,892 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:44,892 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:44,892 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:44,892 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:44,892 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,892 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,893 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,893 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:44,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:44,893 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,893 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:44,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:44,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:44,894 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:44,894 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:44,894 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:44,895 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:44,895 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:44,895 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:44,895 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:44,895 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:44,895 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:44,895 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:44,895 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:44,895 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,895 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:44,895 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:44,896 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,896 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:44,896 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,896 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:44,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:44,897 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:44,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11a2f34a[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:44,897 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,898 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,898 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:44,898 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:44,898 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,898 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:44,898 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:44,899 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:44,899 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:44,899 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,899 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:44,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:44,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:44,901 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:44,901 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:44,901 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:44,901 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:44,901 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:44,902 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,902 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:44,902 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:44,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:44,902 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,903 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:44,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:44,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,905 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:44,905 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:44,905 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:44,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:44,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:44,905 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:44,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:44,905 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,905 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:44,905 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:44,905 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:44,905 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,905 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,905 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:44,905 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:44,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:44,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:44,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:44,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:44,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 407 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693984922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:44,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 416 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37478 deadline: 1682693984932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:44,946 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=688 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/600f6219c5364fb494866015d4394f68 2023-04-28 14:58:44,949 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/9297d81e27c6462bb42b605912f5efe7 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/9297d81e27c6462bb42b605912f5efe7 2023-04-28 14:58:44,952 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/9297d81e27c6462bb42b605912f5efe7, entries=250, sequenceid=688, filesize=39.2 K 2023-04-28 14:58:44,952 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/7bf68f3bd95e4c8982732197b7a03b21 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/7bf68f3bd95e4c8982732197b7a03b21 2023-04-28 14:58:44,955 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/7bf68f3bd95e4c8982732197b7a03b21, entries=150, sequenceid=688, filesize=11.9 K 2023-04-28 14:58:44,956 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/600f6219c5364fb494866015d4394f68 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/600f6219c5364fb494866015d4394f68 2023-04-28 14:58:44,958 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/600f6219c5364fb494866015d4394f68, entries=150, sequenceid=688, filesize=11.9 K 2023-04-28 14:58:44,959 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~161.30 KB/165168, currentSize=127.47 KB/130530 for 1b6356a2d73db55249b3f0cb6bbd108a in 1267ms, sequenceid=688, compaction requested=true 2023-04-28 14:58:44,959 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:44,959 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:44,959 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:58:44,959 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:58:44,959 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:44,959 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:44,961 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 92632 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:58:44,961 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:58:44,961 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): 1b6356a2d73db55249b3f0cb6bbd108a/A is initiating minor compaction (all files) 2023-04-28 14:58:44,961 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 1b6356a2d73db55249b3f0cb6bbd108a/B is initiating minor compaction (all files) 2023-04-28 14:58:44,961 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1b6356a2d73db55249b3f0cb6bbd108a/A in TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:44,961 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/fb08c9f4f01d4d0a93e8c56cdd15eb52, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/fac70c9af9624b5a9934b7aeb3e9b5f6, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/9297d81e27c6462bb42b605912f5efe7] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp, totalSize=90.5 K 2023-04-28 14:58:44,961 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1b6356a2d73db55249b3f0cb6bbd108a/B in TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:44,961 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/daefd9e491814509abb81e77e3fa4b9f, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/3f60af3c093c464187e51ee03191dcde, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/7bf68f3bd95e4c8982732197b7a03b21] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp, totalSize=36.1 K 2023-04-28 14:58:44,961 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting fb08c9f4f01d4d0a93e8c56cdd15eb52, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1682693919678 2023-04-28 14:58:44,962 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting daefd9e491814509abb81e77e3fa4b9f, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1682693919678 2023-04-28 14:58:44,962 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting fac70c9af9624b5a9934b7aeb3e9b5f6, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1682693920322 2023-04-28 14:58:44,962 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 3f60af3c093c464187e51ee03191dcde, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1682693920322 2023-04-28 14:58:44,962 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 9297d81e27c6462bb42b605912f5efe7, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=688, earliestPutTs=1682693923678 2023-04-28 14:58:44,962 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 7bf68f3bd95e4c8982732197b7a03b21, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=688, earliestPutTs=1682693923678 2023-04-28 14:58:44,972 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 1b6356a2d73db55249b3f0cb6bbd108a#B#compaction#327 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:44,975 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1b6356a2d73db55249b3f0cb6bbd108a#A#compaction#328 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:44,988 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/e4e05667d5a94a34af7b01104616c56b as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/e4e05667d5a94a34af7b01104616c56b 2023-04-28 14:58:44,991 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1b6356a2d73db55249b3f0cb6bbd108a/B of 1b6356a2d73db55249b3f0cb6bbd108a into e4e05667d5a94a34af7b01104616c56b(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:44,991 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:44,991 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a., storeName=1b6356a2d73db55249b3f0cb6bbd108a/B, priority=13, startTime=1682693924959; duration=0sec 2023-04-28 14:58:44,991 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:44,991 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:58:44,992 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:58:44,992 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 1b6356a2d73db55249b3f0cb6bbd108a/C is initiating minor compaction (all files) 2023-04-28 14:58:44,992 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1b6356a2d73db55249b3f0cb6bbd108a/C in TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:44,992 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/f7b9295342274723bd36e96bb9ca0f75, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/f3d5f8106ef941a8b561125e7f70d0c9, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/600f6219c5364fb494866015d4394f68] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp, totalSize=36.1 K 2023-04-28 14:58:44,993 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting f7b9295342274723bd36e96bb9ca0f75, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1682693919678 2023-04-28 14:58:44,993 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting f3d5f8106ef941a8b561125e7f70d0c9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1682693920322 2023-04-28 14:58:44,993 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 600f6219c5364fb494866015d4394f68, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=688, earliestPutTs=1682693923678 2023-04-28 14:58:44,999 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 1b6356a2d73db55249b3f0cb6bbd108a#C#compaction#329 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:45,005 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:45,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:45,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:45,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:45,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:45,007 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:45,008 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:45,008 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:45,008 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:45,008 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:45,008 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:45,008 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:45,008 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:45,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:45,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:45,009 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:45,009 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:45,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:45,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:45,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:45,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:45,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:45,009 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:45,009 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:45,009 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:45,009 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:45,009 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:45,009 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:45,010 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:45,010 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:45,010 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:45,010 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1b6356a2d73db55249b3f0cb6bbd108a 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-28 14:58:45,010 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=A 2023-04-28 14:58:45,010 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:45,010 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=B 2023-04-28 14:58:45,010 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:45,010 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=C 2023-04-28 14:58:45,010 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:45,025 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304289ac30abdb6b74fdb99b4ca07f0062b2f to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304289ac30abdb6b74fdb99b4ca07f0062b2f 2023-04-28 14:58:45,405 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/e8fab64420c24b7d9ddfe482f84f6211 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/e8fab64420c24b7d9ddfe482f84f6211 2023-04-28 14:58:45,409 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1b6356a2d73db55249b3f0cb6bbd108a/A of 1b6356a2d73db55249b3f0cb6bbd108a into e8fab64420c24b7d9ddfe482f84f6211(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:45,409 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:45,409 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a., storeName=1b6356a2d73db55249b3f0cb6bbd108a/A, priority=13, startTime=1682693924959; duration=0sec 2023-04-28 14:58:45,409 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:45,410 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/96057a81dd2f4ee38370888b82de2e35 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/96057a81dd2f4ee38370888b82de2e35 2023-04-28 14:58:45,413 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1b6356a2d73db55249b3f0cb6bbd108a/C of 1b6356a2d73db55249b3f0cb6bbd108a into 96057a81dd2f4ee38370888b82de2e35(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:45,414 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:45,414 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a., storeName=1b6356a2d73db55249b3f0cb6bbd108a/C, priority=13, startTime=1682693924959; duration=0sec 2023-04-28 14:58:45,414 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:45,430 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=723, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/43ae9c1ec6d44afc9d4817fe5c8e7b1f 2023-04-28 14:58:45,838 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=723 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/85b11d52e6ba4a5bb01763298286bcf1 2023-04-28 14:58:45,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:45,937 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:45,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:45,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 421 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37478 deadline: 1682693985963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:45,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:45,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 414 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693985966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,072 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:46,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 423 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37478 deadline: 1682693986068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,072 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:46,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 416 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693986068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,247 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=723 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/97fd23d419ae4a8db71f79ad2f984385 2023-04-28 14:58:46,250 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/43ae9c1ec6d44afc9d4817fe5c8e7b1f as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/43ae9c1ec6d44afc9d4817fe5c8e7b1f 2023-04-28 14:58:46,254 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/43ae9c1ec6d44afc9d4817fe5c8e7b1f, entries=150, sequenceid=723, filesize=25.4 K 2023-04-28 14:58:46,254 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/85b11d52e6ba4a5bb01763298286bcf1 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/85b11d52e6ba4a5bb01763298286bcf1 2023-04-28 14:58:46,257 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/85b11d52e6ba4a5bb01763298286bcf1, entries=150, sequenceid=723, filesize=11.9 K 2023-04-28 14:58:46,258 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/97fd23d419ae4a8db71f79ad2f984385 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/97fd23d419ae4a8db71f79ad2f984385 2023-04-28 14:58:46,261 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/97fd23d419ae4a8db71f79ad2f984385, entries=150, sequenceid=723, filesize=11.9 K 2023-04-28 14:58:46,262 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 1b6356a2d73db55249b3f0cb6bbd108a in 1252ms, sequenceid=723, compaction requested=false 2023-04-28 14:58:46,262 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:46,262 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:46,262 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:46,262 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:46,262 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,262 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:46,263 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:46,263 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,263 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:46,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:46,264 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,264 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:46,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:46,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:46,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:46,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:46,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1236466d[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:46,266 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:46,266 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,267 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,267 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:46,267 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:46,267 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:46,267 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,268 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:46,268 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:46,268 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,268 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:46,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:46,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:46,268 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:46,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:46,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:46,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:46,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:46,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:46,270 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:46,270 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:46,270 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:46,270 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:46,270 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:46,271 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,271 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:46,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:46,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,271 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:46,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:46,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:46,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:46,273 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:46,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,273 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:46,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,275 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,275 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:46,275 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:46,275 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:46,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:46,275 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:46,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:46,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:46,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:46,275 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,275 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,275 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,275 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,275 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,275 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:46,275 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:46,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:46,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:46,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:46,277 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1b6356a2d73db55249b3f0cb6bbd108a 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-28 14:58:46,277 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=A 2023-04-28 14:58:46,277 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:46,277 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=B 2023-04-28 14:58:46,277 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:46,277 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=C 2023-04-28 14:58:46,277 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:46,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:46,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 426 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693986311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:46,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 435 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37478 deadline: 1682693986313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,375 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:46,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:46,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:46,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:46,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:46,378 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:46,378 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:46,378 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:46,378 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:46,378 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,378 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,379 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,379 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:46,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:46,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:46,379 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,379 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:46,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:46,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,380 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:46,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:46,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:46,380 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:46,380 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:46,380 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:46,380 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:46,380 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:46,380 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:46,380 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:46,381 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:46,381 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:46,381 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:46,381 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:46,381 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,381 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:46,381 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:46,382 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,382 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:46,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:46,382 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,382 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:46,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:46,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:46,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:46,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:46,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51da1fcd[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:46,383 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:46,383 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,383 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,383 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,384 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:46,384 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:46,384 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:46,384 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:46,384 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,384 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:46,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:46,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:46,384 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:46,384 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:46,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:46,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:46,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:46,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:46,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:46,387 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:46,387 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:46,387 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:46,388 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:46,388 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:46,388 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,388 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,388 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:46,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:46,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:46,389 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:46,390 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:46,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,391 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,391 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:46,391 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,391 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,391 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:46,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:46,391 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,391 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,391 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,391 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:46,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:46,392 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:46,392 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:46,391 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:46,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:46,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:46,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:46,422 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:46,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 428 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693986419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,422 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:46,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 437 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37478 deadline: 1682693986419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,492 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:46,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:46,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:46,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:46,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:46,495 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:46,495 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:46,495 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:46,495 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:46,495 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,495 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,496 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:46,496 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:46,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:46,496 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,496 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:46,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,497 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:46,497 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:46,497 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:46,497 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:46,497 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:46,497 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:46,497 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:46,497 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:46,497 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:46,497 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:46,497 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:46,497 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:46,497 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:46,497 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,497 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:46,497 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:46,498 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,498 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:46,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:46,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:46,498 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,499 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:46,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:46,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:46,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:46,500 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:46,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2620f0f9[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:46,500 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,500 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,500 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:46,500 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:46,500 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,500 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:46,500 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:46,502 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:46,502 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:46,502 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,502 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:46,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:46,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:46,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:46,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:46,504 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:46,504 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:46,504 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:46,504 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:46,504 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:46,505 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,505 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,505 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:46,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:46,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:46,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:46,506 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:46,507 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,508 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:46,508 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:46,508 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:46,508 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:46,508 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:46,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:46,508 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:46,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:46,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:46,508 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,509 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:46,509 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:46,509 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,509 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,509 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,609 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:46,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:46,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:46,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:46,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:46,612 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:46,612 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:46,612 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:46,612 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:46,612 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,612 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,613 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:46,613 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:46,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:46,613 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,614 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:46,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:46,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:46,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:46,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:46,616 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:46,616 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:46,616 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:46,616 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:46,616 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:46,616 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:46,616 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:46,616 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:46,616 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:46,616 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:46,617 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:46,617 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,617 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:46,617 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:46,618 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,618 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:46,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:46,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:46,618 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,618 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:46,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:46,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:46,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:46,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@209e7fd4[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:46,620 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:46,620 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,622 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,622 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:46,622 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:46,622 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:46,622 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,622 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:46,623 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,623 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:46,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:46,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:46,623 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:46,623 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:46,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:46,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:46,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:46,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:46,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:46,626 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:46,626 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:46,626 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:46,626 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:46,626 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:46,627 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,627 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:46,627 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:46,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:46,627 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:46,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:46,628 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:46,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:46,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,629 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:46,629 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:46,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:46,629 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:46,629 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:46,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:46,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:46,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:46,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:46,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:46,629 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,630 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,630 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,630 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,630 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,630 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:46,630 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:46,630 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:46,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:46,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 439 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37478 deadline: 1682693986623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:46,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 430 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693986624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,693 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428bde16a5b1fe74a32a8337f507432fa0f to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428bde16a5b1fe74a32a8337f507432fa0f 2023-04-28 14:58:46,730 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:46,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:46,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:46,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:46,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:46,733 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:46,733 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:46,733 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:46,733 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:46,733 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,733 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,734 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,734 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:46,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:46,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:46,734 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,734 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:46,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:46,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,735 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:46,735 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:46,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:46,735 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:46,735 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:46,735 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:46,735 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:46,735 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:46,735 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:46,735 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:46,736 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:46,736 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:46,736 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:46,736 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:46,736 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,736 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:46,736 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:46,736 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,736 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:46,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:46,737 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,737 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:46,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:46,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:46,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:46,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:46,738 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:46,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bb5084[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:46,738 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,738 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,738 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:46,738 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:46,738 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:46,739 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,739 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:46,739 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,739 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:46,739 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:46,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:46,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:46,739 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:46,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:46,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:46,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:46,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:46,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:46,741 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:46,741 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:46,741 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:46,741 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:46,741 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:46,742 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,742 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,742 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:46,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:46,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:46,742 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:46,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:46,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:46,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:46,744 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:46,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,746 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,746 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:46,746 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,746 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:46,746 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:46,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:46,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:46,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:46,746 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,746 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:46,746 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,746 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:46,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:46,746 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:46,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:46,746 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:46,746 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,846 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:46,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:46,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:46,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:46,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:46,849 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:46,849 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:46,849 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:46,849 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:46,850 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,850 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,850 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,850 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:46,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:46,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:46,851 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,851 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:46,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:46,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:46,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:46,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:46,851 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:46,851 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:46,851 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:46,852 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:46,852 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:46,852 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:46,852 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:46,852 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:46,852 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:46,852 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:46,852 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:46,852 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,852 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:46,852 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:46,852 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,853 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:46,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:46,853 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,853 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:46,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:46,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:46,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:46,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:46,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19e0329[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:46,854 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:46,854 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,854 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,855 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:46,855 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:46,855 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:46,855 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,855 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:46,855 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,855 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:46,855 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:46,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:46,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:46,855 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:46,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:46,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:46,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:46,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:46,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:46,857 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:46,857 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:46,857 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:46,857 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:46,857 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:46,858 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,858 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:46,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:46,858 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:46,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:46,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:46,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:46,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,859 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:46,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,860 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:46,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,862 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:46,862 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:46,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:46,862 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:46,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:46,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:46,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:46,862 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,862 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:46,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:46,862 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,862 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:46,862 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,862 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:46,862 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,862 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,862 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:46,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 432 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693986932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,939 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:46,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 441 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37478 deadline: 1682693986933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,962 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:46,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:46,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:46,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:46,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:46,965 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:46,965 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:46,965 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:46,965 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:46,966 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,966 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,966 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,966 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:46,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:46,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:46,966 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,967 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:46,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:46,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:46,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:46,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:46,968 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:46,968 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:46,968 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:46,968 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:46,968 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:46,968 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:46,968 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:46,968 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:46,968 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:46,968 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:46,968 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:46,968 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,968 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:46,968 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:46,969 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,969 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:46,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:46,969 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,969 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:46,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:46,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:46,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:46,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:46,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20989cdb[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:46,970 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:46,970 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,971 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,971 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:46,971 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:46,971 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:46,971 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,971 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:46,972 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,972 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:46,972 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:46,972 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:46,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:46,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:46,974 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:46,974 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:46,974 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:46,974 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:46,974 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:46,975 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,975 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:46,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:46,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,975 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:46,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:46,975 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:46,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:46,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:46,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,976 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:46,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:46,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:46,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:46,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:46,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:46,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:46,977 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:46,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:46,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:46,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:46,978 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:46,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:46,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:46,978 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:46,978 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:46,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:46,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:47,078 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:47,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:47,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:47,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:47,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:47,080 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:47,080 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:47,081 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:47,081 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:47,081 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:47,081 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:47,082 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:47,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:47,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:47,082 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:47,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:47,083 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:47,083 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:47,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:47,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:47,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:47,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:47,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:47,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:47,084 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:47,084 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:47,084 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:47,084 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:47,084 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:47,084 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:47,085 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:47,085 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:47,085 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:47,085 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:47,085 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:47,085 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:47,085 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:47,085 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:47,086 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:47,086 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:47,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:47,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:47,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:47,086 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:47,086 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:47,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:47,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:47,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:47,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:47,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:47,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:47,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@732b37c3[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:47,090 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:47,090 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:47,091 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:47,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:47,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:47,091 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:47,091 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:47,091 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:47,091 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:47,091 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:47,091 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:47,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:47,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:47,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:47,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:47,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:47,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:47,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:47,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:47,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:47,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:47,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:47,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:47,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:47,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:47,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:47,094 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:47,094 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:47,094 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:47,094 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:47,094 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:47,095 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:47,096 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:47,096 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:47,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:47,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:47,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:47,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:47,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:47,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:47,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:47,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:47,096 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:47,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:47,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:47,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:47,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:47,097 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:47,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:47,098 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=737, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/97748f0de2584c37a6f80957a0e87cad 2023-04-28 14:58:47,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:47,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:47,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:47,100 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:47,100 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:47,100 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:47,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:47,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:47,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:47,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:47,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:47,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:47,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:47,101 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:47,101 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:47,101 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:47,101 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:47,101 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:47,101 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:47,101 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:47,101 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:47,101 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:47,109 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=737 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/998ad5a8d1314c068777efbdf46dc22c 2023-04-28 14:58:47,123 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=737 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/b6fc004b1f954264bc5c55b2962e99be 2023-04-28 14:58:47,127 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/97748f0de2584c37a6f80957a0e87cad as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/97748f0de2584c37a6f80957a0e87cad 2023-04-28 14:58:47,131 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/97748f0de2584c37a6f80957a0e87cad, entries=200, sequenceid=737, filesize=32.3 K 2023-04-28 14:58:47,132 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/998ad5a8d1314c068777efbdf46dc22c as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/998ad5a8d1314c068777efbdf46dc22c 2023-04-28 14:58:47,136 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/998ad5a8d1314c068777efbdf46dc22c, entries=150, sequenceid=737, filesize=11.9 K 2023-04-28 14:58:47,137 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/b6fc004b1f954264bc5c55b2962e99be as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/b6fc004b1f954264bc5c55b2962e99be 2023-04-28 14:58:47,139 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/b6fc004b1f954264bc5c55b2962e99be, entries=150, sequenceid=737, filesize=11.9 K 2023-04-28 14:58:47,140 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 1b6356a2d73db55249b3f0cb6bbd108a in 862ms, sequenceid=737, compaction requested=true 2023-04-28 14:58:47,140 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:47,140 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:47,140 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:47,140 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:58:47,140 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:58:47,140 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:47,141 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85690 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:58:47,141 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:58:47,141 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): 1b6356a2d73db55249b3f0cb6bbd108a/A is initiating minor compaction (all files) 2023-04-28 14:58:47,141 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 1b6356a2d73db55249b3f0cb6bbd108a/B is initiating minor compaction (all files) 2023-04-28 14:58:47,141 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1b6356a2d73db55249b3f0cb6bbd108a/A in TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:47,141 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1b6356a2d73db55249b3f0cb6bbd108a/B in TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:47,141 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/e4e05667d5a94a34af7b01104616c56b, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/85b11d52e6ba4a5bb01763298286bcf1, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/998ad5a8d1314c068777efbdf46dc22c] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp, totalSize=36.2 K 2023-04-28 14:58:47,141 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/e8fab64420c24b7d9ddfe482f84f6211, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/43ae9c1ec6d44afc9d4817fe5c8e7b1f, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/97748f0de2584c37a6f80957a0e87cad] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp, totalSize=83.7 K 2023-04-28 14:58:47,141 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting e4e05667d5a94a34af7b01104616c56b, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=688, earliestPutTs=1682693923678 2023-04-28 14:58:47,141 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting e8fab64420c24b7d9ddfe482f84f6211, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=688, earliestPutTs=1682693923678 2023-04-28 14:58:47,141 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 85b11d52e6ba4a5bb01763298286bcf1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=723, earliestPutTs=1682693923791 2023-04-28 14:58:47,141 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 43ae9c1ec6d44afc9d4817fe5c8e7b1f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=723, earliestPutTs=1682693923791 2023-04-28 14:58:47,141 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 998ad5a8d1314c068777efbdf46dc22c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=737, earliestPutTs=1682693925962 2023-04-28 14:58:47,142 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 97748f0de2584c37a6f80957a0e87cad, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=737, earliestPutTs=1682693925936 2023-04-28 14:58:47,146 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 1b6356a2d73db55249b3f0cb6bbd108a#B#compaction#337 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:47,148 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1b6356a2d73db55249b3f0cb6bbd108a#A#compaction#336 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:47,157 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/d9c406e503b14bf6bae67d72881bba25 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/d9c406e503b14bf6bae67d72881bba25 2023-04-28 14:58:47,160 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1b6356a2d73db55249b3f0cb6bbd108a/B of 1b6356a2d73db55249b3f0cb6bbd108a into d9c406e503b14bf6bae67d72881bba25(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:47,160 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:47,160 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a., storeName=1b6356a2d73db55249b3f0cb6bbd108a/B, priority=13, startTime=1682693927140; duration=0sec 2023-04-28 14:58:47,160 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:47,160 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:58:47,161 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:58:47,161 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 1b6356a2d73db55249b3f0cb6bbd108a/C is initiating minor compaction (all files) 2023-04-28 14:58:47,161 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1b6356a2d73db55249b3f0cb6bbd108a/C in TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:47,161 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/96057a81dd2f4ee38370888b82de2e35, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/97fd23d419ae4a8db71f79ad2f984385, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/b6fc004b1f954264bc5c55b2962e99be] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp, totalSize=36.2 K 2023-04-28 14:58:47,161 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 96057a81dd2f4ee38370888b82de2e35, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=688, earliestPutTs=1682693923678 2023-04-28 14:58:47,161 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 97fd23d419ae4a8db71f79ad2f984385, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=723, earliestPutTs=1682693923791 2023-04-28 14:58:47,162 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting b6fc004b1f954264bc5c55b2962e99be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=737, earliestPutTs=1682693925962 2023-04-28 14:58:47,167 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 1b6356a2d73db55249b3f0cb6bbd108a#C#compaction#338 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:47,201 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:47,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:47,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:47,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:47,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:47,212 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:47,212 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:47,212 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:47,213 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:47,213 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:47,213 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:47,214 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:47,214 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:47,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:47,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:47,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:47,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:47,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:47,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:47,215 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:47,215 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:47,215 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:47,215 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:47,215 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:47,215 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:47,216 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:47,216 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:47,218 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:47,218 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:47,218 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:47,218 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:47,218 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1b6356a2d73db55249b3f0cb6bbd108a 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-28 14:58:47,218 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=A 2023-04-28 14:58:47,219 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:47,219 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=B 2023-04-28 14:58:47,219 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:47,219 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=C 2023-04-28 14:58:47,219 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:47,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:47,444 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:47,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:47,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 447 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37478 deadline: 1682693987474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:47,482 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:47,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 439 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693987474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:47,579 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/d0332e2cc43e488286ea496f96a0454a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/d0332e2cc43e488286ea496f96a0454a 2023-04-28 14:58:47,587 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1b6356a2d73db55249b3f0cb6bbd108a/C of 1b6356a2d73db55249b3f0cb6bbd108a into d0332e2cc43e488286ea496f96a0454a(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:47,587 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:47,587 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a., storeName=1b6356a2d73db55249b3f0cb6bbd108a/C, priority=13, startTime=1682693927140; duration=0sec 2023-04-28 14:58:47,587 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:47,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:47,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 441 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693987584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:47,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:47,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 449 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37478 deadline: 1682693987585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:47,673 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428a974dd36d28644d485c993215eae5503 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a974dd36d28644d485c993215eae5503 2023-04-28 14:58:47,700 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=758, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/dad8b2582ec1463ea5ba2603c707e534 2023-04-28 14:58:47,796 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:47,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 443 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693987790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:47,796 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:47,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 451 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37478 deadline: 1682693987793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:47,996 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/f343dfedaf33435885b0e9a22c8b4d4f as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/f343dfedaf33435885b0e9a22c8b4d4f 2023-04-28 14:58:47,999 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1b6356a2d73db55249b3f0cb6bbd108a/A of 1b6356a2d73db55249b3f0cb6bbd108a into f343dfedaf33435885b0e9a22c8b4d4f(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:47,999 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:48,000 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a., storeName=1b6356a2d73db55249b3f0cb6bbd108a/A, priority=13, startTime=1682693927140; duration=0sec 2023-04-28 14:58:48,000 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:48,081 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:48,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37454 deadline: 1682693988080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:48,083 DEBUG [Thread-2107] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18225 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a., hostname=jenkins-hbase17.apache.org,37181,1682693812482, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 14:58:48,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:48,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37450 deadline: 1682693988096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:48,100 DEBUG [Thread-2105] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18241 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a., hostname=jenkins-hbase17.apache.org,37181,1682693812482, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 14:58:48,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:48,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 453 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37478 deadline: 1682693988098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:48,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:48,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 445 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693988100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:48,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:48,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:37496 deadline: 1682693988131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:48,135 DEBUG [Thread-2103] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18277 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a., hostname=jenkins-hbase17.apache.org,37181,1682693812482, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 14:58:48,135 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=758 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/752a61696b434a908bf45bac3c75265c 2023-04-28 14:58:48,143 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=758 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/8ea27e03b7a74f4bab35c68fb670938a 2023-04-28 14:58:48,146 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/dad8b2582ec1463ea5ba2603c707e534 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/dad8b2582ec1463ea5ba2603c707e534 2023-04-28 14:58:48,150 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/dad8b2582ec1463ea5ba2603c707e534, entries=150, sequenceid=758, filesize=25.4 K 2023-04-28 14:58:48,151 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/752a61696b434a908bf45bac3c75265c as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/752a61696b434a908bf45bac3c75265c 2023-04-28 14:58:48,153 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/752a61696b434a908bf45bac3c75265c, entries=150, sequenceid=758, filesize=11.9 K 2023-04-28 14:58:48,154 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/8ea27e03b7a74f4bab35c68fb670938a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/8ea27e03b7a74f4bab35c68fb670938a 2023-04-28 14:58:48,157 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/8ea27e03b7a74f4bab35c68fb670938a, entries=150, sequenceid=758, filesize=11.9 K 2023-04-28 14:58:48,158 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 1b6356a2d73db55249b3f0cb6bbd108a in 940ms, sequenceid=758, compaction requested=false 2023-04-28 14:58:48,158 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:48,158 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:48,158 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:48,158 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:48,158 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:48,158 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:48,158 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:48,159 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:48,159 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:48,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:48,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:48,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:48,159 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:48,159 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:48,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:48,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:48,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:48,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:48,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:48,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:48,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29ce7652[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:48,161 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:48,161 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:48,162 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:48,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:48,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:48,162 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:48,162 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:48,162 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:48,162 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:48,162 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:48,163 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:48,163 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:48,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:48,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:48,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:48,163 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:48,163 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:48,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:48,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:48,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:48,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:48,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:48,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:48,170 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:48,170 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:48,170 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:48,170 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:48,170 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:48,171 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:48,171 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:48,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:48,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:48,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:48,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:48,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:48,171 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:48,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:48,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:48,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:48,174 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:48,175 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:48,177 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:48,177 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:48,177 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:48,177 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:48,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:48,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:48,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:48,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:48,177 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:48,177 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:48,177 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:48,178 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:48,178 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:48,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:48,178 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:48,178 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:48,178 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:48,278 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:48,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:48,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:58:48,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:58:48,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:58:48,281 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:58:48,281 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:48,281 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:58:48,281 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:58:48,281 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:48,281 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:48,286 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:48,286 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:48,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:48,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:48,287 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:48,287 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:58:48,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:48,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:48,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:58:48,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:48,288 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:58:48,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:58:48,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:58:48,288 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:58:48,288 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:58:48,288 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:58:48,288 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:58:48,288 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:58:48,288 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:48,288 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. started... 2023-04-28 14:58:48,288 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1b6356a2d73db55249b3f0cb6bbd108a 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-28 14:58:48,288 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=A 2023-04-28 14:58:48,289 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:48,289 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=B 2023-04-28 14:58:48,289 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:48,289 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=C 2023-04-28 14:58:48,289 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:48,307 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428a97d4bf1ba4c47b89bc11add3162ff54 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a97d4bf1ba4c47b89bc11add3162ff54 2023-04-28 14:58:48,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:48,611 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. as already flushing 2023-04-28 14:58:48,689 DEBUG [Thread-2114] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x14d061d2 to 127.0.0.1:61762 2023-04-28 14:58:48,689 DEBUG [Thread-2114] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:58:48,693 DEBUG [Thread-2112] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ded66ad to 127.0.0.1:61762 2023-04-28 14:58:48,693 DEBUG [Thread-2112] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:58:48,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:48,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 456 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693988682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:48,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:48,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 463 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37478 deadline: 1682693988682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:48,695 DEBUG [Thread-2118] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x143a1290 to 127.0.0.1:61762 2023-04-28 14:58:48,695 DEBUG [Thread-2118] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:58:48,697 DEBUG [Thread-2116] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x228ff1a4 to 127.0.0.1:61762 2023-04-28 14:58:48,697 DEBUG [Thread-2116] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:58:48,697 DEBUG [Thread-2110] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x36a9ece3 to 127.0.0.1:61762 2023-04-28 14:58:48,698 DEBUG [Thread-2110] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:58:48,713 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=772, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/50f6b1058e3b408bade828ecec0ea553 2023-04-28 14:58:48,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:48,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:48,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 458 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693988795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:48,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 465 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37478 deadline: 1682693988795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:48,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:48,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:58:48,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 467 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37478 deadline: 1682693988998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:48,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 460 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:37424 deadline: 1682693988998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:49,124 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=772 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/264ef2ca86914d629d217a118eac5d28 2023-04-28 14:58:49,142 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=772 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/77c654790dcd44de89293a5c616fa03d 2023-04-28 14:58:49,149 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/50f6b1058e3b408bade828ecec0ea553 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/50f6b1058e3b408bade828ecec0ea553 2023-04-28 14:58:49,155 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/50f6b1058e3b408bade828ecec0ea553, entries=100, sequenceid=772, filesize=18.6 K 2023-04-28 14:58:49,156 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/264ef2ca86914d629d217a118eac5d28 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/264ef2ca86914d629d217a118eac5d28 2023-04-28 14:58:49,160 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/264ef2ca86914d629d217a118eac5d28, entries=100, sequenceid=772, filesize=9.5 K 2023-04-28 14:58:49,161 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/77c654790dcd44de89293a5c616fa03d as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/77c654790dcd44de89293a5c616fa03d 2023-04-28 14:58:49,165 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/77c654790dcd44de89293a5c616fa03d, entries=100, sequenceid=772, filesize=9.5 K 2023-04-28 14:58:49,166 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 1b6356a2d73db55249b3f0cb6bbd108a in 878ms, sequenceid=772, compaction requested=true 2023-04-28 14:58:49,166 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:49,166 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:49,166 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:58:49,166 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:58:49,166 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:49,166 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:58:49,166 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:58:49,168 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:49,168 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:49,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:49,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:49,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:49,168 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:49,168 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:58:49,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:49,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:49,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:58:49,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58b8af09[Count = 0] remaining members to acquire global barrier 2023-04-28 14:58:49,170 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:58:49,170 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:49,171 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:49,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:49,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:49,171 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:58:49,171 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:58:49,172 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:58:49,172 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:49,172 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:58:49,173 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:49,173 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:58:49,173 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:49,173 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:58:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:49,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:49,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:49,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:49,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:49,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:49,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:49,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:58:49,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:58:49,177 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:58:49,177 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:58:49,177 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:58:49,177 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:58:49,177 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:58:49,179 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:49,179 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:49,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:49,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:49,179 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:49,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:49,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:58:49,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:58:49,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:49,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:49,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:58:49,180 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:49,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:49,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:49,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:58:49,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:49,182 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:49,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:49,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:58:49,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:58:49,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:49,184 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:49,184 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:58:49,185 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:49,185 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:58:49,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:58:49,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:58:49,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:58:49,185 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:58:49,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:58:49,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:58:49,185 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:58:49,185 DEBUG [Thread-2109] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:58:49,185 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:49,186 DEBUG [Thread-2109] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:58:49,186 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:49,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:58:49,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:58:49,186 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:58:49,186 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:58:49,286 DEBUG [Thread-2109] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:58:49,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:58:49,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:49,302 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1b6356a2d73db55249b3f0cb6bbd108a 3/3 column families, dataSize=127.47 KB heapSize=379.97 KB 2023-04-28 14:58:49,302 DEBUG [Thread-2099] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x74a355de to 127.0.0.1:61762 2023-04-28 14:58:49,302 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=A 2023-04-28 14:58:49,302 DEBUG [Thread-2101] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x57b6628e to 127.0.0.1:61762 2023-04-28 14:58:49,303 DEBUG [Thread-2101] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:58:49,302 DEBUG [Thread-2099] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:58:49,303 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:49,303 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=B 2023-04-28 14:58:49,304 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:49,304 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=C 2023-04-28 14:58:49,304 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:58:49,725 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304284037590ed35d4a94a9b195482017dbd1 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304284037590ed35d4a94a9b195482017dbd1 2023-04-28 14:58:50,131 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=794, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/91b5f18c22134838baf82ec9b6f600a2 2023-04-28 14:58:50,549 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=794 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/674c8757a44e46adbd77f42518fef122 2023-04-28 14:58:50,967 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=794 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/3f65b466042942e7ad810d53229d4c39 2023-04-28 14:58:50,974 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/91b5f18c22134838baf82ec9b6f600a2 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/91b5f18c22134838baf82ec9b6f600a2 2023-04-28 14:58:50,980 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/91b5f18c22134838baf82ec9b6f600a2, entries=150, sequenceid=794, filesize=25.4 K 2023-04-28 14:58:50,981 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/674c8757a44e46adbd77f42518fef122 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/674c8757a44e46adbd77f42518fef122 2023-04-28 14:58:50,987 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/674c8757a44e46adbd77f42518fef122, entries=150, sequenceid=794, filesize=11.9 K 2023-04-28 14:58:50,988 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/3f65b466042942e7ad810d53229d4c39 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/3f65b466042942e7ad810d53229d4c39 2023-04-28 14:58:50,993 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/3f65b466042942e7ad810d53229d4c39, entries=150, sequenceid=794, filesize=11.9 K 2023-04-28 14:58:50,994 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~379.92 KB/389040, currentSize=0 B/0 for 1b6356a2d73db55249b3f0cb6bbd108a in 1692ms, sequenceid=794, compaction requested=true 2023-04-28 14:58:50,994 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:50,994 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:50,994 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:58:50,994 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:58:50,994 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:50,994 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:50,996 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 97750 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:58:50,996 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46914 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:58:50,996 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): 1b6356a2d73db55249b3f0cb6bbd108a/A is initiating minor compaction (all files) 2023-04-28 14:58:50,996 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 1b6356a2d73db55249b3f0cb6bbd108a/B is initiating minor compaction (all files) 2023-04-28 14:58:50,996 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1b6356a2d73db55249b3f0cb6bbd108a/A in TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:50,996 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1b6356a2d73db55249b3f0cb6bbd108a/B in TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:50,996 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/f343dfedaf33435885b0e9a22c8b4d4f, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/dad8b2582ec1463ea5ba2603c707e534, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/50f6b1058e3b408bade828ecec0ea553, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/91b5f18c22134838baf82ec9b6f600a2] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp, totalSize=95.5 K 2023-04-28 14:58:50,996 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/d9c406e503b14bf6bae67d72881bba25, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/752a61696b434a908bf45bac3c75265c, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/264ef2ca86914d629d217a118eac5d28, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/674c8757a44e46adbd77f42518fef122] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp, totalSize=45.8 K 2023-04-28 14:58:50,997 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting f343dfedaf33435885b0e9a22c8b4d4f, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=737, earliestPutTs=1682693925962 2023-04-28 14:58:50,997 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting d9c406e503b14bf6bae67d72881bba25, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=737, earliestPutTs=1682693925962 2023-04-28 14:58:50,998 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting dad8b2582ec1463ea5ba2603c707e534, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=758, earliestPutTs=1682693926306 2023-04-28 14:58:50,998 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 752a61696b434a908bf45bac3c75265c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=758, earliestPutTs=1682693926306 2023-04-28 14:58:50,998 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 50f6b1058e3b408bade828ecec0ea553, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=772, earliestPutTs=1682693927461 2023-04-28 14:58:50,998 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 264ef2ca86914d629d217a118eac5d28, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=772, earliestPutTs=1682693927461 2023-04-28 14:58:50,998 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 91b5f18c22134838baf82ec9b6f600a2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=794, earliestPutTs=1682693928679 2023-04-28 14:58:50,998 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 674c8757a44e46adbd77f42518fef122, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=794, earliestPutTs=1682693928679 2023-04-28 14:58:51,005 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 1b6356a2d73db55249b3f0cb6bbd108a#B#compaction#348 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:51,006 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1b6356a2d73db55249b3f0cb6bbd108a#A#compaction#349 average throughput is 9.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:51,419 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/ee960e142bd84a0cbb3ba295b4bd5843 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/ee960e142bd84a0cbb3ba295b4bd5843 2023-04-28 14:58:51,423 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1b6356a2d73db55249b3f0cb6bbd108a/B of 1b6356a2d73db55249b3f0cb6bbd108a into ee960e142bd84a0cbb3ba295b4bd5843(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:51,423 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:51,423 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a., storeName=1b6356a2d73db55249b3f0cb6bbd108a/B, priority=12, startTime=1682693930994; duration=0sec 2023-04-28 14:58:51,423 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:58:51,424 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:58:51,425 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46914 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:58:51,425 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 1b6356a2d73db55249b3f0cb6bbd108a/C is initiating minor compaction (all files) 2023-04-28 14:58:51,425 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1b6356a2d73db55249b3f0cb6bbd108a/C in TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:51,425 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/d0332e2cc43e488286ea496f96a0454a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/8ea27e03b7a74f4bab35c68fb670938a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/77c654790dcd44de89293a5c616fa03d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/3f65b466042942e7ad810d53229d4c39] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp, totalSize=45.8 K 2023-04-28 14:58:51,425 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting d0332e2cc43e488286ea496f96a0454a, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=737, earliestPutTs=1682693925962 2023-04-28 14:58:51,426 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 8ea27e03b7a74f4bab35c68fb670938a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=758, earliestPutTs=1682693926306 2023-04-28 14:58:51,426 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 77c654790dcd44de89293a5c616fa03d, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=772, earliestPutTs=1682693927461 2023-04-28 14:58:51,426 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 3f65b466042942e7ad810d53229d4c39, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=794, earliestPutTs=1682693928679 2023-04-28 14:58:51,432 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 1b6356a2d73db55249b3f0cb6bbd108a#C#compaction#350 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:58:51,837 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/ed56e8aa16ba4b9f8ba6efab4d122106 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/ed56e8aa16ba4b9f8ba6efab4d122106 2023-04-28 14:58:51,842 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/93af708560a14025a0f1a27a12a7f149 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/93af708560a14025a0f1a27a12a7f149 2023-04-28 14:58:51,843 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1b6356a2d73db55249b3f0cb6bbd108a/A of 1b6356a2d73db55249b3f0cb6bbd108a into ed56e8aa16ba4b9f8ba6efab4d122106(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:51,843 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:51,843 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a., storeName=1b6356a2d73db55249b3f0cb6bbd108a/A, priority=12, startTime=1682693930994; duration=0sec 2023-04-28 14:58:51,844 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:51,847 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1b6356a2d73db55249b3f0cb6bbd108a/C of 1b6356a2d73db55249b3f0cb6bbd108a into 93af708560a14025a0f1a27a12a7f149(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:58:51,847 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:58:51,847 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a., storeName=1b6356a2d73db55249b3f0cb6bbd108a/C, priority=12, startTime=1682693930994; duration=0sec 2023-04-28 14:58:51,847 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:58:53,711 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/8cc291db9d1e4b5aaf853dc2ebbb8272, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/2139eba17656443bb9439982b35222a4, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/98f5ebaf27084bcdb60741015cf813ea, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/1fe914c2bc8c4b53ba41f590ff5c3504, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/904d6b2d9cf44200a3f6a70cee5ec6e1, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/1e5211d6c9be47d2af90f35ae1d19606, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/ee9b85658f1943e7b84b049563af6afe, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/d45bf5f28f9f4c299e1d92de9c65c639, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/3bb39b3d938f45bca45ed1c7c5c33b1b, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/d886632b0c194a4e8edc48db9c3b0da5, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/9af92d3dfb304e17bc935f50388c0840, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/fe328da62af74718957712487a1c53ab, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/fb08c9f4f01d4d0a93e8c56cdd15eb52, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/fac70c9af9624b5a9934b7aeb3e9b5f6, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/9297d81e27c6462bb42b605912f5efe7, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/e8fab64420c24b7d9ddfe482f84f6211, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/43ae9c1ec6d44afc9d4817fe5c8e7b1f, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/97748f0de2584c37a6f80957a0e87cad, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/f343dfedaf33435885b0e9a22c8b4d4f, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/dad8b2582ec1463ea5ba2603c707e534, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/50f6b1058e3b408bade828ecec0ea553, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/91b5f18c22134838baf82ec9b6f600a2] to archive 2023-04-28 14:58:53,712 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 14:58:53,713 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/8cc291db9d1e4b5aaf853dc2ebbb8272 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/8cc291db9d1e4b5aaf853dc2ebbb8272 2023-04-28 14:58:53,714 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/2139eba17656443bb9439982b35222a4 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/2139eba17656443bb9439982b35222a4 2023-04-28 14:58:53,715 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/98f5ebaf27084bcdb60741015cf813ea to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/98f5ebaf27084bcdb60741015cf813ea 2023-04-28 14:58:53,716 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/1fe914c2bc8c4b53ba41f590ff5c3504 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/1fe914c2bc8c4b53ba41f590ff5c3504 2023-04-28 14:58:53,717 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/904d6b2d9cf44200a3f6a70cee5ec6e1 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/904d6b2d9cf44200a3f6a70cee5ec6e1 2023-04-28 14:58:53,718 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/1e5211d6c9be47d2af90f35ae1d19606 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/1e5211d6c9be47d2af90f35ae1d19606 2023-04-28 14:58:53,719 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/ee9b85658f1943e7b84b049563af6afe to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/ee9b85658f1943e7b84b049563af6afe 2023-04-28 14:58:53,720 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/d45bf5f28f9f4c299e1d92de9c65c639 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/d45bf5f28f9f4c299e1d92de9c65c639 2023-04-28 14:58:53,721 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/3bb39b3d938f45bca45ed1c7c5c33b1b to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/3bb39b3d938f45bca45ed1c7c5c33b1b 2023-04-28 14:58:53,722 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/d886632b0c194a4e8edc48db9c3b0da5 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/d886632b0c194a4e8edc48db9c3b0da5 2023-04-28 14:58:53,723 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/9af92d3dfb304e17bc935f50388c0840 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/9af92d3dfb304e17bc935f50388c0840 2023-04-28 14:58:53,724 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/fe328da62af74718957712487a1c53ab to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/fe328da62af74718957712487a1c53ab 2023-04-28 14:58:53,724 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/fb08c9f4f01d4d0a93e8c56cdd15eb52 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/fb08c9f4f01d4d0a93e8c56cdd15eb52 2023-04-28 14:58:53,725 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/fac70c9af9624b5a9934b7aeb3e9b5f6 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/fac70c9af9624b5a9934b7aeb3e9b5f6 2023-04-28 14:58:53,726 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/9297d81e27c6462bb42b605912f5efe7 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/9297d81e27c6462bb42b605912f5efe7 2023-04-28 14:58:53,727 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/e8fab64420c24b7d9ddfe482f84f6211 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/e8fab64420c24b7d9ddfe482f84f6211 2023-04-28 14:58:53,728 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/43ae9c1ec6d44afc9d4817fe5c8e7b1f to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/43ae9c1ec6d44afc9d4817fe5c8e7b1f 2023-04-28 14:58:53,729 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/97748f0de2584c37a6f80957a0e87cad to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/97748f0de2584c37a6f80957a0e87cad 2023-04-28 14:58:53,730 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/f343dfedaf33435885b0e9a22c8b4d4f to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/f343dfedaf33435885b0e9a22c8b4d4f 2023-04-28 14:58:53,731 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/dad8b2582ec1463ea5ba2603c707e534 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/dad8b2582ec1463ea5ba2603c707e534 2023-04-28 14:58:53,732 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/50f6b1058e3b408bade828ecec0ea553 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/50f6b1058e3b408bade828ecec0ea553 2023-04-28 14:58:53,733 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/91b5f18c22134838baf82ec9b6f600a2 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/91b5f18c22134838baf82ec9b6f600a2 2023-04-28 14:58:53,739 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/f47036c560f64891946b5871cfadbc26, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/0aca0e5ea7744233b8271ba9f1ad98b6, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/76adf38f24534e2e85b1d5c370dcda3e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/7139dae4f3a847f6983d39f5795f45a7, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/74201117f358422a9bc72df0fede463a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/2a9b9c1d30a641ae8df30685f492177d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/e947743dfcbf4590a7a29276fae44ea4, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/50e702134ae74dd6884303929eb10233, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/0babcce5a49e4211adb6c01503200b41, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/83181b7025a24918b3869a25655eb522, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/4ce3f94d2e0842c5a6142964a664ee2f, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/daefd9e491814509abb81e77e3fa4b9f, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/d11afaf9a1fc4548b1ec0dd7a3ce858f, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/3f60af3c093c464187e51ee03191dcde, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/e4e05667d5a94a34af7b01104616c56b, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/7bf68f3bd95e4c8982732197b7a03b21, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/85b11d52e6ba4a5bb01763298286bcf1, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/d9c406e503b14bf6bae67d72881bba25, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/998ad5a8d1314c068777efbdf46dc22c, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/752a61696b434a908bf45bac3c75265c, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/264ef2ca86914d629d217a118eac5d28, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/674c8757a44e46adbd77f42518fef122] to archive 2023-04-28 14:58:53,740 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 14:58:53,741 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/f47036c560f64891946b5871cfadbc26 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/f47036c560f64891946b5871cfadbc26 2023-04-28 14:58:53,742 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/0aca0e5ea7744233b8271ba9f1ad98b6 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/0aca0e5ea7744233b8271ba9f1ad98b6 2023-04-28 14:58:53,743 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/76adf38f24534e2e85b1d5c370dcda3e to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/76adf38f24534e2e85b1d5c370dcda3e 2023-04-28 14:58:53,744 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/7139dae4f3a847f6983d39f5795f45a7 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/7139dae4f3a847f6983d39f5795f45a7 2023-04-28 14:58:53,745 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/74201117f358422a9bc72df0fede463a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/74201117f358422a9bc72df0fede463a 2023-04-28 14:58:53,746 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/2a9b9c1d30a641ae8df30685f492177d to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/2a9b9c1d30a641ae8df30685f492177d 2023-04-28 14:58:53,747 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/e947743dfcbf4590a7a29276fae44ea4 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/e947743dfcbf4590a7a29276fae44ea4 2023-04-28 14:58:53,748 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/50e702134ae74dd6884303929eb10233 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/50e702134ae74dd6884303929eb10233 2023-04-28 14:58:53,748 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/0babcce5a49e4211adb6c01503200b41 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/0babcce5a49e4211adb6c01503200b41 2023-04-28 14:58:53,749 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/83181b7025a24918b3869a25655eb522 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/83181b7025a24918b3869a25655eb522 2023-04-28 14:58:53,750 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/4ce3f94d2e0842c5a6142964a664ee2f to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/4ce3f94d2e0842c5a6142964a664ee2f 2023-04-28 14:58:53,751 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/daefd9e491814509abb81e77e3fa4b9f to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/daefd9e491814509abb81e77e3fa4b9f 2023-04-28 14:58:53,752 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/d11afaf9a1fc4548b1ec0dd7a3ce858f to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/d11afaf9a1fc4548b1ec0dd7a3ce858f 2023-04-28 14:58:53,752 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/3f60af3c093c464187e51ee03191dcde to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/3f60af3c093c464187e51ee03191dcde 2023-04-28 14:58:53,753 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/e4e05667d5a94a34af7b01104616c56b to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/e4e05667d5a94a34af7b01104616c56b 2023-04-28 14:58:53,754 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/7bf68f3bd95e4c8982732197b7a03b21 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/7bf68f3bd95e4c8982732197b7a03b21 2023-04-28 14:58:53,755 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/85b11d52e6ba4a5bb01763298286bcf1 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/85b11d52e6ba4a5bb01763298286bcf1 2023-04-28 14:58:53,755 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/d9c406e503b14bf6bae67d72881bba25 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/d9c406e503b14bf6bae67d72881bba25 2023-04-28 14:58:53,756 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/998ad5a8d1314c068777efbdf46dc22c to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/998ad5a8d1314c068777efbdf46dc22c 2023-04-28 14:58:53,757 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/752a61696b434a908bf45bac3c75265c to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/752a61696b434a908bf45bac3c75265c 2023-04-28 14:58:53,758 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/264ef2ca86914d629d217a118eac5d28 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/264ef2ca86914d629d217a118eac5d28 2023-04-28 14:58:53,759 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/674c8757a44e46adbd77f42518fef122 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/674c8757a44e46adbd77f42518fef122 2023-04-28 14:58:53,762 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/b70b8d11e45945879fa01ae59008878c, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/a230028051b94adfa5a13ef6f5b9052e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/684c5fd877d34127b5f57b210c1a1e91, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/17b70c903b714e83a2fdab80eaa83098, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/2d50b616bc2b41d1bf2836b544db4943, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/52fd3a4b770b4489b78a838d85796a05, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/cf3ba797893847148b398d7d16af6573, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/f660052b0d1045f7893cdfcd2d5ee1b4, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/f71e25b0a19744d893aaaf6a2ea37d2e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/064f1cf2d5d24f8782a4640695f1b519, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/c21b06c7d9af40ada1ff3cd7f572c583, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/f7b9295342274723bd36e96bb9ca0f75, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/65829dd1d4d24cddabccacc16243210d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/f3d5f8106ef941a8b561125e7f70d0c9, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/96057a81dd2f4ee38370888b82de2e35, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/600f6219c5364fb494866015d4394f68, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/97fd23d419ae4a8db71f79ad2f984385, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/d0332e2cc43e488286ea496f96a0454a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/b6fc004b1f954264bc5c55b2962e99be, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/8ea27e03b7a74f4bab35c68fb670938a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/77c654790dcd44de89293a5c616fa03d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/3f65b466042942e7ad810d53229d4c39] to archive 2023-04-28 14:58:53,763 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 14:58:53,764 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/b70b8d11e45945879fa01ae59008878c to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/b70b8d11e45945879fa01ae59008878c 2023-04-28 14:58:53,765 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/a230028051b94adfa5a13ef6f5b9052e to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/a230028051b94adfa5a13ef6f5b9052e 2023-04-28 14:58:53,766 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/684c5fd877d34127b5f57b210c1a1e91 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/684c5fd877d34127b5f57b210c1a1e91 2023-04-28 14:58:53,767 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/17b70c903b714e83a2fdab80eaa83098 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/17b70c903b714e83a2fdab80eaa83098 2023-04-28 14:58:53,768 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/2d50b616bc2b41d1bf2836b544db4943 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/2d50b616bc2b41d1bf2836b544db4943 2023-04-28 14:58:53,769 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/52fd3a4b770b4489b78a838d85796a05 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/52fd3a4b770b4489b78a838d85796a05 2023-04-28 14:58:53,770 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/cf3ba797893847148b398d7d16af6573 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/cf3ba797893847148b398d7d16af6573 2023-04-28 14:58:53,771 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/f660052b0d1045f7893cdfcd2d5ee1b4 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/f660052b0d1045f7893cdfcd2d5ee1b4 2023-04-28 14:58:53,772 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/f71e25b0a19744d893aaaf6a2ea37d2e to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/f71e25b0a19744d893aaaf6a2ea37d2e 2023-04-28 14:58:53,773 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/064f1cf2d5d24f8782a4640695f1b519 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/064f1cf2d5d24f8782a4640695f1b519 2023-04-28 14:58:53,774 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/c21b06c7d9af40ada1ff3cd7f572c583 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/c21b06c7d9af40ada1ff3cd7f572c583 2023-04-28 14:58:53,775 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/f7b9295342274723bd36e96bb9ca0f75 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/f7b9295342274723bd36e96bb9ca0f75 2023-04-28 14:58:53,777 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/65829dd1d4d24cddabccacc16243210d to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/65829dd1d4d24cddabccacc16243210d 2023-04-28 14:58:53,778 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/f3d5f8106ef941a8b561125e7f70d0c9 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/f3d5f8106ef941a8b561125e7f70d0c9 2023-04-28 14:58:53,780 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/96057a81dd2f4ee38370888b82de2e35 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/96057a81dd2f4ee38370888b82de2e35 2023-04-28 14:58:53,781 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/600f6219c5364fb494866015d4394f68 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/600f6219c5364fb494866015d4394f68 2023-04-28 14:58:53,782 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/97fd23d419ae4a8db71f79ad2f984385 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/97fd23d419ae4a8db71f79ad2f984385 2023-04-28 14:58:53,784 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/d0332e2cc43e488286ea496f96a0454a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/d0332e2cc43e488286ea496f96a0454a 2023-04-28 14:58:53,785 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/b6fc004b1f954264bc5c55b2962e99be to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/b6fc004b1f954264bc5c55b2962e99be 2023-04-28 14:58:53,786 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/8ea27e03b7a74f4bab35c68fb670938a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/8ea27e03b7a74f4bab35c68fb670938a 2023-04-28 14:58:53,787 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/77c654790dcd44de89293a5c616fa03d to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/77c654790dcd44de89293a5c616fa03d 2023-04-28 14:58:53,787 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/3f65b466042942e7ad810d53229d4c39 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/3f65b466042942e7ad810d53229d4c39 2023-04-28 14:58:58,102 DEBUG [Thread-2107] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x060c5cba to 127.0.0.1:61762 2023-04-28 14:58:58,102 DEBUG [Thread-2107] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:58:58,156 DEBUG [Thread-2105] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x74d6bf18 to 127.0.0.1:61762 2023-04-28 14:58:58,156 DEBUG [Thread-2105] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:58:58,177 DEBUG [Thread-2103] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5da5030e to 127.0.0.1:61762 2023-04-28 14:58:58,177 DEBUG [Thread-2103] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:58:58,177 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-28 14:58:58,177 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(394): wrote 356 2023-04-28 14:58:58,177 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(394): wrote 347 2023-04-28 14:58:58,178 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(394): wrote 8 2023-04-28 14:58:58,178 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(394): wrote 7 2023-04-28 14:58:58,178 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(394): wrote 9 2023-04-28 14:58:58,178 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-28 14:58:58,178 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-28 14:58:58,178 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(402): scanned 2116 2023-04-28 14:58:58,178 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(403): verified 6342 rows 2023-04-28 14:58:58,178 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(402): scanned 2100 2023-04-28 14:58:58,178 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(403): verified 6300 rows 2023-04-28 14:58:58,179 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(402): scanned 2122 2023-04-28 14:58:58,179 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(403): verified 6359 rows 2023-04-28 14:58:58,179 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(402): scanned 2106 2023-04-28 14:58:58,179 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(403): verified 6317 rows 2023-04-28 14:58:58,179 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(402): scanned 2115 2023-04-28 14:58:58,179 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(403): verified 6344 rows 2023-04-28 14:58:58,179 INFO [Listener at localhost.localdomain/40193] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-28 14:58:58,180 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7d243e46 to 127.0.0.1:61762 2023-04-28 14:58:58,180 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:58:58,182 INFO [Listener at localhost.localdomain/40193] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-28 14:58:58,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.HMaster$11(2418): Client=jenkins//136.243.18.41 disable TestAcidGuarantees 2023-04-28 14:58:58,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-28 14:58:58,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-28 14:58:58,188 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682693938188"}]},"ts":"1682693938188"} 2023-04-28 14:58:58,189 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-28 14:58:58,189 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-28 14:58:58,190 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1b6356a2d73db55249b3f0cb6bbd108a, UNASSIGN}] 2023-04-28 14:58:58,190 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1b6356a2d73db55249b3f0cb6bbd108a, UNASSIGN 2023-04-28 14:58:58,191 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=1b6356a2d73db55249b3f0cb6bbd108a, regionState=CLOSING, regionLocation=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:58:58,191 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693938191"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682693938191"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682693938191"}]},"ts":"1682693938191"} 2023-04-28 14:58:58,192 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482}] 2023-04-28 14:58:58,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-28 14:58:58,344 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(109): Close 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:58:58,345 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1604): Closing 1b6356a2d73db55249b3f0cb6bbd108a, disabling compactions & flushes 2023-04-28 14:58:58,345 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:58,345 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:58,345 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. after waiting 0 ms 2023-04-28 14:58:58,345 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:58:58,345 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2745): Flushing 1b6356a2d73db55249b3f0cb6bbd108a 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-28 14:58:58,346 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=A 2023-04-28 14:58:58,346 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:58,346 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=B 2023-04-28 14:58:58,347 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:58,347 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1b6356a2d73db55249b3f0cb6bbd108a, store=C 2023-04-28 14:58:58,347 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:58:58,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-28 14:58:58,771 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304286e7138e03716441fb6e2023d94b32450 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304286e7138e03716441fb6e2023d94b32450 2023-04-28 14:58:58,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-28 14:58:59,179 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=803, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/149a93565a4b4075ae8601c51a308f7a 2023-04-28 14:58:59,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-28 14:58:59,599 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=803 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/c1fb9ba5188f45d998126ea091eb309d 2023-04-28 14:59:00,011 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=803 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/404c6d797817477586fb9a6b770d8b5c 2023-04-28 14:59:00,018 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/A/149a93565a4b4075ae8601c51a308f7a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/149a93565a4b4075ae8601c51a308f7a 2023-04-28 14:59:00,023 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/149a93565a4b4075ae8601c51a308f7a, entries=100, sequenceid=803, filesize=18.6 K 2023-04-28 14:59:00,025 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/B/c1fb9ba5188f45d998126ea091eb309d as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/c1fb9ba5188f45d998126ea091eb309d 2023-04-28 14:59:00,030 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/c1fb9ba5188f45d998126ea091eb309d, entries=100, sequenceid=803, filesize=9.5 K 2023-04-28 14:59:00,031 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/.tmp/C/404c6d797817477586fb9a6b770d8b5c as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/404c6d797817477586fb9a6b770d8b5c 2023-04-28 14:59:00,037 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/404c6d797817477586fb9a6b770d8b5c, entries=100, sequenceid=803, filesize=9.5 K 2023-04-28 14:59:00,037 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 1b6356a2d73db55249b3f0cb6bbd108a in 1692ms, sequenceid=803, compaction requested=false 2023-04-28 14:59:00,051 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/recovered.edits/806.seqid, newMaxSeqId=806, maxSeqId=4 2023-04-28 14:59:00,053 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a. 2023-04-28 14:59:00,053 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1558): Region close journal for 1b6356a2d73db55249b3f0cb6bbd108a: 2023-04-28 14:59:00,055 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(147): Closed 1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:59:00,056 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=1b6356a2d73db55249b3f0cb6bbd108a, regionState=CLOSED 2023-04-28 14:59:00,056 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693940056"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682693940056"}]},"ts":"1682693940056"} 2023-04-28 14:59:00,059 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-28 14:59:00,059 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 1b6356a2d73db55249b3f0cb6bbd108a, server=jenkins-hbase17.apache.org,37181,1682693812482 in 1.8650 sec 2023-04-28 14:59:00,061 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-28 14:59:00,061 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1b6356a2d73db55249b3f0cb6bbd108a, UNASSIGN in 1.8690 sec 2023-04-28 14:59:00,061 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682693940061"}]},"ts":"1682693940061"} 2023-04-28 14:59:00,062 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-28 14:59:00,063 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-28 14:59:00,065 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.8800 sec 2023-04-28 14:59:00,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-28 14:59:00,296 INFO [Listener at localhost.localdomain/40193] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-28 14:59:00,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.HMaster$5(2228): Client=jenkins//136.243.18.41 delete TestAcidGuarantees 2023-04-28 14:59:00,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:59:00,300 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:59:00,301 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:59:00,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-28 14:59:00,308 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:59:00,312 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A, FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B, FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C, FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/recovered.edits] 2023-04-28 14:59:00,317 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/149a93565a4b4075ae8601c51a308f7a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/149a93565a4b4075ae8601c51a308f7a 2023-04-28 14:59:00,318 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/ed56e8aa16ba4b9f8ba6efab4d122106 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/A/ed56e8aa16ba4b9f8ba6efab4d122106 2023-04-28 14:59:00,319 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/c1fb9ba5188f45d998126ea091eb309d to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/c1fb9ba5188f45d998126ea091eb309d 2023-04-28 14:59:00,320 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/ee960e142bd84a0cbb3ba295b4bd5843 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/B/ee960e142bd84a0cbb3ba295b4bd5843 2023-04-28 14:59:00,322 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/404c6d797817477586fb9a6b770d8b5c to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/404c6d797817477586fb9a6b770d8b5c 2023-04-28 14:59:00,323 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/93af708560a14025a0f1a27a12a7f149 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/C/93af708560a14025a0f1a27a12a7f149 2023-04-28 14:59:00,325 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/recovered.edits/806.seqid to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a/recovered.edits/806.seqid 2023-04-28 14:59:00,325 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/1b6356a2d73db55249b3f0cb6bbd108a 2023-04-28 14:59:00,325 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 14:59:00,326 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-28 14:59:00,326 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-28 14:59:00,328 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042808b69a7b574f456ea1f20e25f775867f to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042808b69a7b574f456ea1f20e25f775867f 2023-04-28 14:59:00,330 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042808fb6ff7e23340b6bff30eb8df776141 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042808fb6ff7e23340b6bff30eb8df776141 2023-04-28 14:59:00,331 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304281eece70ae65644b9adea49454716aab7 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304281eece70ae65644b9adea49454716aab7 2023-04-28 14:59:00,332 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304284037590ed35d4a94a9b195482017dbd1 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304284037590ed35d4a94a9b195482017dbd1 2023-04-28 14:59:00,333 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304286e7138e03716441fb6e2023d94b32450 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304286e7138e03716441fb6e2023d94b32450 2023-04-28 14:59:00,334 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428770d9ccc6c7e4d0d9895af12d656737c to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428770d9ccc6c7e4d0d9895af12d656737c 2023-04-28 14:59:00,335 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042885a015f1fae44e19ab95e85bb20377a3 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042885a015f1fae44e19ab95e85bb20377a3 2023-04-28 14:59:00,336 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042898708376ae794ce3a08874348464d6ed to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042898708376ae794ce3a08874348464d6ed 2023-04-28 14:59:00,337 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304289ac30abdb6b74fdb99b4ca07f0062b2f to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304289ac30abdb6b74fdb99b4ca07f0062b2f 2023-04-28 14:59:00,338 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a974dd36d28644d485c993215eae5503 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a974dd36d28644d485c993215eae5503 2023-04-28 14:59:00,339 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a97d4bf1ba4c47b89bc11add3162ff54 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a97d4bf1ba4c47b89bc11add3162ff54 2023-04-28 14:59:00,340 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428adb2f4d4065e469d886d2762c701ff9f to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428adb2f4d4065e469d886d2762c701ff9f 2023-04-28 14:59:00,341 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428bde16a5b1fe74a32a8337f507432fa0f to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428bde16a5b1fe74a32a8337f507432fa0f 2023-04-28 14:59:00,342 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428c3a26e6d8e124eaabceba5b74cab9678 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428c3a26e6d8e124eaabceba5b74cab9678 2023-04-28 14:59:00,343 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428de824d32214d4b52be7cc10f18b27b46 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428de824d32214d4b52be7cc10f18b27b46 2023-04-28 14:59:00,344 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428e612a12b266f4d88bd9fd38086e9a472 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428e612a12b266f4d88bd9fd38086e9a472 2023-04-28 14:59:00,345 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ea8d17ea802e4885b8875cc686a5567f to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ea8d17ea802e4885b8875cc686a5567f 2023-04-28 14:59:00,346 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428f4648bf067384a32ab0dad90d3a3a485 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428f4648bf067384a32ab0dad90d3a3a485 2023-04-28 14:59:00,347 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-28 14:59:00,348 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:59:00,349 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-28 14:59:00,350 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-28 14:59:00,351 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:59:00,351 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-28 14:59:00,351 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682693940351"}]},"ts":"9223372036854775807"} 2023-04-28 14:59:00,352 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-28 14:59:00,352 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 1b6356a2d73db55249b3f0cb6bbd108a, NAME => 'TestAcidGuarantees,,1682693906578.1b6356a2d73db55249b3f0cb6bbd108a.', STARTKEY => '', ENDKEY => ''}] 2023-04-28 14:59:00,352 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-28 14:59:00,352 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682693940352"}]},"ts":"9223372036854775807"} 2023-04-28 14:59:00,353 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-28 14:59:00,354 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:59:00,355 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 57 msec 2023-04-28 14:59:00,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-28 14:59:00,403 INFO [Listener at localhost.localdomain/40193] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-28 14:59:00,420 INFO [Listener at localhost.localdomain/40193] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227 (was 227), OpenFileDescriptor=529 (was 566), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=334 (was 364), ProcessCount=168 (was 168), AvailableMemoryMB=4972 (was 4925) - AvailableMemoryMB LEAK? - 2023-04-28 14:59:00,431 INFO [Listener at localhost.localdomain/40193] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227, OpenFileDescriptor=529, MaxFileDescriptor=60000, SystemLoadAverage=334, ProcessCount=168, AvailableMemoryMB=4971 2023-04-28 14:59:00,433 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-28 14:59:00,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.HMaster$4(2112): Client=jenkins//136.243.18.41 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-28 14:59:00,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-28 14:59:00,436 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-28 14:59:00,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(697): Client=jenkins//136.243.18.41 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-28 14:59:00,437 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-28 14:59:00,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-28 14:59:00,437 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:00,438 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714 empty. 2023-04-28 14:59:00,438 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:00,438 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 14:59:00,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-28 14:59:00,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-28 14:59:00,846 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-28 14:59:00,848 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 7fd2108ac9c35550edc28d90fafe4714, NAME => 'TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp 2023-04-28 14:59:01,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-28 14:59:01,257 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 14:59:01,257 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 7fd2108ac9c35550edc28d90fafe4714, disabling compactions & flushes 2023-04-28 14:59:01,257 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:01,257 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:01,257 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. after waiting 0 ms 2023-04-28 14:59:01,257 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:01,257 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:01,257 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:01,261 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-28 14:59:01,262 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693941262"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682693941262"}]},"ts":"1682693941262"} 2023-04-28 14:59:01,263 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-28 14:59:01,263 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-28 14:59:01,264 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682693941263"}]},"ts":"1682693941263"} 2023-04-28 14:59:01,264 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-28 14:59:01,266 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7fd2108ac9c35550edc28d90fafe4714, ASSIGN}] 2023-04-28 14:59:01,267 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7fd2108ac9c35550edc28d90fafe4714, ASSIGN 2023-04-28 14:59:01,268 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=7fd2108ac9c35550edc28d90fafe4714, ASSIGN; state=OFFLINE, location=jenkins-hbase17.apache.org,37181,1682693812482; forceNewPlan=false, retain=false 2023-04-28 14:59:01,419 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=7fd2108ac9c35550edc28d90fafe4714, regionState=OPENING, regionLocation=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:01,419 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693941419"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682693941419"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682693941419"}]},"ts":"1682693941419"} 2023-04-28 14:59:01,420 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482}] 2023-04-28 14:59:01,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-28 14:59:01,575 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:01,575 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7fd2108ac9c35550edc28d90fafe4714, NAME => 'TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.', STARTKEY => '', ENDKEY => ''} 2023-04-28 14:59:01,575 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:01,575 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 14:59:01,576 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7894): checking encryption for 7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:01,576 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7897): checking classloading for 7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:01,577 INFO [StoreOpener-7fd2108ac9c35550edc28d90fafe4714-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:01,578 DEBUG [StoreOpener-7fd2108ac9c35550edc28d90fafe4714-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A 2023-04-28 14:59:01,578 DEBUG [StoreOpener-7fd2108ac9c35550edc28d90fafe4714-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A 2023-04-28 14:59:01,578 INFO [StoreOpener-7fd2108ac9c35550edc28d90fafe4714-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 14:59:01,578 INFO [StoreOpener-7fd2108ac9c35550edc28d90fafe4714-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7fd2108ac9c35550edc28d90fafe4714 columnFamilyName A 2023-04-28 14:59:01,578 INFO [StoreOpener-7fd2108ac9c35550edc28d90fafe4714-1] regionserver.HStore(310): Store=7fd2108ac9c35550edc28d90fafe4714/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:59:01,578 INFO [StoreOpener-7fd2108ac9c35550edc28d90fafe4714-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:01,580 DEBUG [StoreOpener-7fd2108ac9c35550edc28d90fafe4714-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B 2023-04-28 14:59:01,580 DEBUG [StoreOpener-7fd2108ac9c35550edc28d90fafe4714-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B 2023-04-28 14:59:01,580 INFO [StoreOpener-7fd2108ac9c35550edc28d90fafe4714-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 14:59:01,580 INFO [StoreOpener-7fd2108ac9c35550edc28d90fafe4714-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7fd2108ac9c35550edc28d90fafe4714 columnFamilyName B 2023-04-28 14:59:01,580 INFO [StoreOpener-7fd2108ac9c35550edc28d90fafe4714-1] regionserver.HStore(310): Store=7fd2108ac9c35550edc28d90fafe4714/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:59:01,580 INFO [StoreOpener-7fd2108ac9c35550edc28d90fafe4714-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:01,581 DEBUG [StoreOpener-7fd2108ac9c35550edc28d90fafe4714-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C 2023-04-28 14:59:01,581 DEBUG [StoreOpener-7fd2108ac9c35550edc28d90fafe4714-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C 2023-04-28 14:59:01,582 INFO [StoreOpener-7fd2108ac9c35550edc28d90fafe4714-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 14:59:01,582 INFO [StoreOpener-7fd2108ac9c35550edc28d90fafe4714-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7fd2108ac9c35550edc28d90fafe4714 columnFamilyName C 2023-04-28 14:59:01,582 INFO [StoreOpener-7fd2108ac9c35550edc28d90fafe4714-1] regionserver.HStore(310): Store=7fd2108ac9c35550edc28d90fafe4714/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:59:01,582 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:01,583 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:01,583 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:01,585 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 14:59:01,585 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1055): writing seq id for 7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:01,587 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-28 14:59:01,587 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1072): Opened 7fd2108ac9c35550edc28d90fafe4714; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59922314, jitterRate=-0.10708793997764587}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 14:59:01,587 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(965): Region open journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:01,588 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714., pid=49, masterSystemTime=1682693941572 2023-04-28 14:59:01,589 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:01,589 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:01,589 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=7fd2108ac9c35550edc28d90fafe4714, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:01,589 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693941589"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1682693941589"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682693941589"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682693941589"}]},"ts":"1682693941589"} 2023-04-28 14:59:01,590 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-28 14:59:01,591 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 in 170 msec 2023-04-28 14:59:01,592 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-28 14:59:01,592 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=7fd2108ac9c35550edc28d90fafe4714, ASSIGN in 325 msec 2023-04-28 14:59:01,592 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-28 14:59:01,592 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682693941592"}]},"ts":"1682693941592"} 2023-04-28 14:59:01,593 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-28 14:59:01,594 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-28 14:59:01,595 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1600 sec 2023-04-28 14:59:02,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-28 14:59:02,546 INFO [Listener at localhost.localdomain/40193] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-28 14:59:02,547 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x1a885842 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:59:02,557 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@9b9721, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:59:02,559 DEBUG [Listener at localhost.localdomain/40193] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:59:02,561 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:52036, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:59:02,562 DEBUG [Listener at localhost.localdomain/40193] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-28 14:59:02,564 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:44628, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-28 14:59:02,565 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x06846250 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:59:02,570 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2cb7a788, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:59:02,570 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x647f95f6 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:59:02,575 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@41ccedcb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:59:02,575 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x58ae18e2 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:59:02,580 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@40e6f624, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:59:02,581 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x029c1e76 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:59:02,585 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2a5d7fcf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:59:02,586 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x2c240146 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:59:02,591 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4ea9d51a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:59:02,592 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x31fd1c1f to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:59:02,597 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@31fc0e3f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:59:02,598 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x209faa3c to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:59:02,602 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@79db57d4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:59:02,603 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x5345748a to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:59:02,607 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@e56ea94, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:59:02,607 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x3c9f61fa to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:59:02,612 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4486b3c7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:59:02,612 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x534826c2 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:59:02,617 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@149f1de0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:59:02,628 DEBUG [hconnection-0x88b7896-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:59:02,629 DEBUG [hconnection-0x55caa554-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:59:02,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:02,629 DEBUG [hconnection-0x181cfc4c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:59:02,629 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:52058, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:59:02,631 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:52046, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:59:02,631 DEBUG [hconnection-0x2543633e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:59:02,631 DEBUG [hconnection-0x1771ce82-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:59:02,632 DEBUG [hconnection-0x72701385-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:59:02,632 DEBUG [hconnection-0x46d7c617-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:59:02,632 DEBUG [hconnection-0x2f791f8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:59:02,636 DEBUG [hconnection-0x5eb88ebc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:59:02,636 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:52082, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:59:02,636 DEBUG [hconnection-0x2196d6bf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:59:02,636 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:52068, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:59:02,637 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:52118, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:59:02,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:02,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:02,642 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:02,642 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:02,642 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:02,642 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:02,642 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:52100, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:59:02,642 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:52088, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:59:02,643 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:52124, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:59:02,643 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:52104, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:59:02,643 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:52136, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:59:02,643 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:02,647 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:02,649 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:02,649 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:02,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:02,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:02,650 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:02,650 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:02,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:02,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:02,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:02,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:02,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:02,651 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:02,651 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:02,651 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:02,651 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:02,651 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:02,651 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:02,651 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:02,651 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:02,652 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:02,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:02,655 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:02,655 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7fd2108ac9c35550edc28d90fafe4714 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-28 14:59:02,658 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=A 2023-04-28 14:59:02,658 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:02,658 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=B 2023-04-28 14:59:02,658 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:02,658 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=C 2023-04-28 14:59:02,658 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:02,669 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/8c095c95c9ef48a3acdd6c8b91e07e4f 2023-04-28 14:59:02,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:02,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694002673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:02,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:02,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 4 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694002673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:02,678 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:02,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694002673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:02,678 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:02,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694002674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:02,678 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:02,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694002674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:02,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:02,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694002776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:02,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:02,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694002778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:02,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:02,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694002778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:02,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:02,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694002779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:02,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:02,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694002779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:02,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:02,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694002982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:02,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:02,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694002982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:02,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:02,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694002982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:02,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:02,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694002984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:02,987 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:02,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694002984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,084 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/54c4e7ddbe614602a11ecd19702be32e 2023-04-28 14:59:03,291 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:03,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694003287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,291 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:03,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694003288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,292 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:03,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694003288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,292 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:03,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694003289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,292 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:03,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694003289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,492 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/04ac0565409b45fd9c061f1a6eb3d7f7 2023-04-28 14:59:03,496 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/8c095c95c9ef48a3acdd6c8b91e07e4f as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/8c095c95c9ef48a3acdd6c8b91e07e4f 2023-04-28 14:59:03,498 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/8c095c95c9ef48a3acdd6c8b91e07e4f, entries=150, sequenceid=13, filesize=11.6 K 2023-04-28 14:59:03,499 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/54c4e7ddbe614602a11ecd19702be32e as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/54c4e7ddbe614602a11ecd19702be32e 2023-04-28 14:59:03,504 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/54c4e7ddbe614602a11ecd19702be32e, entries=150, sequenceid=13, filesize=11.6 K 2023-04-28 14:59:03,505 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/04ac0565409b45fd9c061f1a6eb3d7f7 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/04ac0565409b45fd9c061f1a6eb3d7f7 2023-04-28 14:59:03,508 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/04ac0565409b45fd9c061f1a6eb3d7f7, entries=150, sequenceid=13, filesize=11.6 K 2023-04-28 14:59:03,508 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 7fd2108ac9c35550edc28d90fafe4714 in 853ms, sequenceid=13, compaction requested=false 2023-04-28 14:59:03,508 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:03,509 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:03,509 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:03,509 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:03,509 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,509 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:03,509 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:03,509 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,509 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:03,510 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:03,510 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:03,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:03,511 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:03,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ca29e81[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:03,511 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:03,511 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:03,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:03,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:03,512 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:03,512 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:03,512 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:03,512 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,512 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:03,512 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:03,512 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,512 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:03,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:03,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:03,512 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:03,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:03,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:03,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:03,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:03,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:03,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:03,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:03,515 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:03,515 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:03,515 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:03,515 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:03,515 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:03,516 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:03,516 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:03,516 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:03,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:03,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:03,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:03,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:03,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:03,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:03,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:03,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:03,516 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:03,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:03,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:03,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:03,517 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:03,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:03,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,519 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:03,519 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:03,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:03,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:03,519 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,519 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:03,519 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:03,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:03,519 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:03,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:03,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:03,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:03,519 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,520 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:03,520 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:03,520 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:03,520 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:03,520 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:03,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:03,620 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:03,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:03,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:03,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:03,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:03,623 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:03,623 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:03,623 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:03,623 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:03,623 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:03,623 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:03,624 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:03,624 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:03,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:03,624 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,624 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:03,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:03,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:03,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:03,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:03,624 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:03,625 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:03,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:03,625 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:03,625 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:03,625 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:03,625 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:03,625 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:03,625 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:03,625 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:03,625 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7fd2108ac9c35550edc28d90fafe4714 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-28 14:59:03,626 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=A 2023-04-28 14:59:03,626 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:03,626 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=B 2023-04-28 14:59:03,626 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:03,626 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=C 2023-04-28 14:59:03,626 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:03,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:03,795 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:03,808 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:03,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694003801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,808 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:03,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694003802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,809 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:03,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694003803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,809 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:03,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694003804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:03,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694003808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:03,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694003909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:03,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694003909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:03,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694003909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:03,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694003909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:03,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:03,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694003911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,036 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/ee2438536df941fb984ac1ff91cafc01 2023-04-28 14:59:04,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:04,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694004111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,112 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:04,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694004111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,112 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:04,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694004111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,112 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:04,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694004111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:04,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694004117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:04,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694004413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:04,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694004413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:04,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694004414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,417 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:04,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694004414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:04,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694004422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,443 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/0e2e1d9ec6714bd0beaf28981711369e 2023-04-28 14:59:04,853 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/fd212d398664432cbe673de037a969c5 2023-04-28 14:59:04,855 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/ee2438536df941fb984ac1ff91cafc01 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/ee2438536df941fb984ac1ff91cafc01 2023-04-28 14:59:04,858 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/ee2438536df941fb984ac1ff91cafc01, entries=150, sequenceid=33, filesize=11.6 K 2023-04-28 14:59:04,859 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/0e2e1d9ec6714bd0beaf28981711369e as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/0e2e1d9ec6714bd0beaf28981711369e 2023-04-28 14:59:04,862 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/0e2e1d9ec6714bd0beaf28981711369e, entries=150, sequenceid=33, filesize=11.6 K 2023-04-28 14:59:04,863 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/fd212d398664432cbe673de037a969c5 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/fd212d398664432cbe673de037a969c5 2023-04-28 14:59:04,865 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/fd212d398664432cbe673de037a969c5, entries=150, sequenceid=33, filesize=11.6 K 2023-04-28 14:59:04,866 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 7fd2108ac9c35550edc28d90fafe4714 in 1241ms, sequenceid=33, compaction requested=false 2023-04-28 14:59:04,866 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:04,866 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:04,866 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:04,866 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:04,866 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,866 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:04,866 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:04,867 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,867 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:04,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:04,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:04,867 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:04,867 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:04,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:04,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:04,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:04,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:04,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:04,868 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:04,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@161d23f9[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:04,868 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:04,869 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:04,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:04,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:04,869 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:04,869 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:04,869 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:04,869 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,869 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:04,869 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:04,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:04,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:04,870 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:04,870 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:04,870 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:04,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:04,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:04,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:04,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:04,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:04,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:04,872 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:04,872 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:04,872 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:04,872 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:04,872 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:04,873 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:04,873 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:04,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:04,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:04,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:04,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:04,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:04,873 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:04,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:04,873 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:04,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:04,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:04,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:04,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:04,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:04,874 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:04,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:04,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,875 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,875 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:04,875 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:04,875 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:04,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:04,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:04,875 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:04,875 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:04,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:04,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:04,875 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:04,876 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:04,876 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:04,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:04,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:04,876 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:04,876 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:04,876 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:04,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:04,923 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7fd2108ac9c35550edc28d90fafe4714 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-28 14:59:04,923 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=A 2023-04-28 14:59:04,923 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:04,923 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=B 2023-04-28 14:59:04,923 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:04,923 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=C 2023-04-28 14:59:04,923 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:04,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:04,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694004960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,974 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/b8840ab117f44adaae6ba92611caa7cd 2023-04-28 14:59:04,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:04,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694004963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,975 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:04,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694004965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,975 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:04,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694004966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,975 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:04,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694004974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,976 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:04,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:04,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:04,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:04,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:04,979 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:04,979 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:04,979 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:04,979 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:04,979 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:04,979 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:04,980 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,980 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:04,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:04,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:04,980 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,980 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:04,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:04,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:04,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:04,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:04,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:04,982 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:04,982 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:04,982 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:04,982 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:04,982 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:04,986 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:04,986 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:04,986 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:04,986 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:04,986 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:04,986 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:04,986 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:04,986 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:04,986 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,989 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:04,989 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:04,990 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:04,990 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:04,990 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:04,991 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:04,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:04,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:04,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:04,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:04,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:04,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:04,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cbbbb8e[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:04,993 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:04,993 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:04,993 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:04,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:04,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:04,993 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:04,993 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:04,993 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:04,994 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,994 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:04,994 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,994 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:04,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:04,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:04,994 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:04,994 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:04,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:04,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:04,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:04,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:04,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:04,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:04,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:04,997 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:04,997 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:04,997 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:04,997 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:04,997 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:04,998 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:04,998 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:04,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:04,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:04,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:04,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:04,998 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:04,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:04,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:04,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:04,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:04,999 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:04,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:04,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:04,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:05,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:05,000 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:05,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:05,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,002 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:05,002 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:05,002 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,002 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:05,002 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:05,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:05,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:05,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:05,002 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:05,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:05,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:05,003 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,003 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:05,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:05,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:05,003 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:05,003 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:05,003 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:05,003 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:05,016 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/c874a8d494144edda1cc7ab6c13eb14a 2023-04-28 14:59:05,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:05,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694005075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:05,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694005077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,084 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:05,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694005077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,084 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:05,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694005078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,084 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:05,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694005078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,104 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:05,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:05,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:05,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:05,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:05,114 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:05,114 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:05,114 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:05,114 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:05,115 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:05,115 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:05,116 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,116 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,116 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:05,116 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:05,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:05,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:05,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:05,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:05,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:05,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:05,118 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:05,119 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:05,119 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:05,119 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:05,119 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:05,119 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:05,122 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:05,122 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:05,122 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:05,122 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:05,122 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:05,122 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:05,122 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:05,122 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:05,122 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,122 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:05,122 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:05,127 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:05,127 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:05,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:05,127 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:05,127 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:05,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:05,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:05,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:05,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:05,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:05,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51e685f7[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:05,131 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:05,131 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:05,131 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:05,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:05,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:05,132 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:05,132 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:05,132 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:05,134 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,134 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:05,136 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,136 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:05,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:05,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:05,136 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:05,136 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:05,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:05,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:05,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:05,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:05,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:05,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:05,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:05,139 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:05,139 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:05,139 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:05,139 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:05,139 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:05,140 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:05,140 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:05,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:05,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:05,141 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:05,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:05,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:05,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:05,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:05,141 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:05,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:05,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:05,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:05,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:05,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:05,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:05,144 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:05,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,146 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,147 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:05,147 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:05,147 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:05,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:05,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:05,147 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:05,147 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,147 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:05,147 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:05,147 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:05,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:05,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:05,147 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:05,147 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:05,147 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:05,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:05,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:05,247 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:05,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:05,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:05,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:05,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:05,250 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:05,250 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:05,250 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:05,250 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:05,250 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:05,250 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:05,251 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,251 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:05,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:05,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:05,251 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,251 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:05,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:05,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:05,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:05,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:05,252 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:05,254 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:05,254 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:05,254 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:05,254 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:05,254 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:05,254 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:05,254 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:05,254 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:05,254 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:05,254 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:05,254 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:05,254 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:05,255 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:05,255 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,255 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:05,255 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:05,256 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:05,256 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:05,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:05,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:05,256 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:05,256 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:05,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:05,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:05,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:05,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:05,257 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:05,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79984fdc[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:05,257 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:05,257 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:05,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:05,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:05,258 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:05,258 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:05,258 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:05,258 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,258 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:05,258 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,258 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:05,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:05,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:05,258 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:05,258 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:05,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:05,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:05,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:05,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:05,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:05,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:05,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:05,262 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:05,262 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:05,262 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:05,262 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:05,262 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:05,263 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:05,263 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:05,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:05,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:05,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:05,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:05,263 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:05,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:05,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:05,263 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:05,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:05,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:05,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:05,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:05,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:05,264 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:05,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:05,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,265 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:05,265 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,265 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:05,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:05,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:05,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:05,266 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:05,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:05,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:05,265 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:05,267 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:05,267 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:05,267 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:05,267 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,267 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:05,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:05,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:05,267 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:05,267 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:05,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:05,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694005278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:05,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694005285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:05,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694005285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:05,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694005285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,287 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:05,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694005285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,367 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:05,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:05,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:05,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:05,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:05,370 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:05,370 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:05,370 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:05,370 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:05,370 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:05,371 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:05,371 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,371 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:05,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:05,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:05,371 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,371 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:05,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:05,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:05,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:05,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:05,372 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:05,372 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:05,372 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:05,372 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:05,372 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:05,372 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:05,372 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:05,372 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:05,372 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:05,372 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:05,373 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:05,373 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:05,373 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:05,373 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:05,373 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,373 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:05,373 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:05,373 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,373 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:05,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:05,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:05,374 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:05,374 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:05,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:05,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:05,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:05,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:05,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:05,375 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:05,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@665a9fd[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:05,375 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:05,375 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:05,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:05,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:05,376 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:05,376 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:05,376 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,376 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:05,376 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:05,376 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,376 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:05,376 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:05,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:05,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:05,376 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:05,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:05,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:05,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:05,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:05,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:05,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:05,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:05,378 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:05,378 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:05,378 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:05,378 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:05,378 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:05,379 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:05,379 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:05,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:05,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:05,379 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:05,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:05,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:05,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:05,380 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:05,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:05,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:05,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:05,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:05,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:05,381 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:05,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:05,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:05,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,382 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,382 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:05,382 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:05,382 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:05,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:05,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:05,382 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:05,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:05,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:05,382 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:05,383 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,383 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:05,383 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:05,383 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:05,383 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:05,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:05,383 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:05,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:05,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:05,443 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/2be99a8c1cef412389390bca57fcd486 2023-04-28 14:59:05,446 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/b8840ab117f44adaae6ba92611caa7cd as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/b8840ab117f44adaae6ba92611caa7cd 2023-04-28 14:59:05,452 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/b8840ab117f44adaae6ba92611caa7cd, entries=150, sequenceid=46, filesize=11.6 K 2023-04-28 14:59:05,453 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/c874a8d494144edda1cc7ab6c13eb14a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/c874a8d494144edda1cc7ab6c13eb14a 2023-04-28 14:59:05,456 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/c874a8d494144edda1cc7ab6c13eb14a, entries=150, sequenceid=46, filesize=11.6 K 2023-04-28 14:59:05,456 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/2be99a8c1cef412389390bca57fcd486 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/2be99a8c1cef412389390bca57fcd486 2023-04-28 14:59:05,459 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/2be99a8c1cef412389390bca57fcd486, entries=150, sequenceid=46, filesize=11.6 K 2023-04-28 14:59:05,460 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 7fd2108ac9c35550edc28d90fafe4714 in 537ms, sequenceid=46, compaction requested=true 2023-04-28 14:59:05,460 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:05,460 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:05,460 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:59:05,460 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:59:05,460 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:05,460 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:05,461 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:59:05,461 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): 7fd2108ac9c35550edc28d90fafe4714/A is initiating minor compaction (all files) 2023-04-28 14:59:05,461 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7fd2108ac9c35550edc28d90fafe4714/A in TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:05,461 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/8c095c95c9ef48a3acdd6c8b91e07e4f, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/ee2438536df941fb984ac1ff91cafc01, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/b8840ab117f44adaae6ba92611caa7cd] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp, totalSize=34.8 K 2023-04-28 14:59:05,461 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 8c095c95c9ef48a3acdd6c8b91e07e4f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1682693942651 2023-04-28 14:59:05,461 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting ee2438536df941fb984ac1ff91cafc01, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1682693942670 2023-04-28 14:59:05,462 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting b8840ab117f44adaae6ba92611caa7cd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1682693943801 2023-04-28 14:59:05,467 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:59:05,467 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 7fd2108ac9c35550edc28d90fafe4714/B is initiating minor compaction (all files) 2023-04-28 14:59:05,467 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7fd2108ac9c35550edc28d90fafe4714/B in TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:05,467 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/54c4e7ddbe614602a11ecd19702be32e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/0e2e1d9ec6714bd0beaf28981711369e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/c874a8d494144edda1cc7ab6c13eb14a] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp, totalSize=34.8 K 2023-04-28 14:59:05,467 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 54c4e7ddbe614602a11ecd19702be32e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1682693942651 2023-04-28 14:59:05,468 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 0e2e1d9ec6714bd0beaf28981711369e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1682693942670 2023-04-28 14:59:05,469 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting c874a8d494144edda1cc7ab6c13eb14a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1682693943801 2023-04-28 14:59:05,475 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7fd2108ac9c35550edc28d90fafe4714#A#compaction#363 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:59:05,483 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:05,483 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 7fd2108ac9c35550edc28d90fafe4714#B#compaction#364 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:59:05,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:05,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:05,519 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/3109db1b49e3434497771ef287f1faa9 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/3109db1b49e3434497771ef287f1faa9 2023-04-28 14:59:05,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:05,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:05,522 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:05,522 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:05,522 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:05,522 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:05,522 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:05,522 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:05,524 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,524 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:05,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:05,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:05,524 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,524 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:05,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:05,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:05,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:05,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:05,525 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:05,527 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/a6a105000c874b78b6788e28ae1ad3c7 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/a6a105000c874b78b6788e28ae1ad3c7 2023-04-28 14:59:05,531 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7fd2108ac9c35550edc28d90fafe4714/B of 7fd2108ac9c35550edc28d90fafe4714 into a6a105000c874b78b6788e28ae1ad3c7(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:05,531 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:05,531 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714., storeName=7fd2108ac9c35550edc28d90fafe4714/B, priority=13, startTime=1682693945460; duration=0sec 2023-04-28 14:59:05,531 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:05,531 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:59:05,532 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:59:05,532 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 7fd2108ac9c35550edc28d90fafe4714/C is initiating minor compaction (all files) 2023-04-28 14:59:05,532 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7fd2108ac9c35550edc28d90fafe4714/C in TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:05,532 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/04ac0565409b45fd9c061f1a6eb3d7f7, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/fd212d398664432cbe673de037a969c5, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/2be99a8c1cef412389390bca57fcd486] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp, totalSize=34.8 K 2023-04-28 14:59:05,532 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 04ac0565409b45fd9c061f1a6eb3d7f7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1682693942651 2023-04-28 14:59:05,532 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting fd212d398664432cbe673de037a969c5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1682693942670 2023-04-28 14:59:05,533 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 2be99a8c1cef412389390bca57fcd486, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1682693943801 2023-04-28 14:59:05,534 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:05,534 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:05,534 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:05,534 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:05,534 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:05,535 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7fd2108ac9c35550edc28d90fafe4714/A of 7fd2108ac9c35550edc28d90fafe4714 into 3109db1b49e3434497771ef287f1faa9(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:05,535 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:05,535 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714., storeName=7fd2108ac9c35550edc28d90fafe4714/A, priority=13, startTime=1682693945460; duration=0sec 2023-04-28 14:59:05,535 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:05,535 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:05,535 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:05,535 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:05,535 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:05,535 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7fd2108ac9c35550edc28d90fafe4714 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-28 14:59:05,536 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=A 2023-04-28 14:59:05,536 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:05,536 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=B 2023-04-28 14:59:05,536 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:05,536 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=C 2023-04-28 14:59:05,536 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:05,539 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 7fd2108ac9c35550edc28d90fafe4714#C#compaction#365 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:59:05,584 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-28 14:59:05,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:05,585 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:05,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:05,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694005596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:05,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694005597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:05,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694005598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:05,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694005599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:05,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694005600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:05,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694005703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:05,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694005703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:05,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694005704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:05,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:05,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694005704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694005704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,908 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:05,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694005905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,908 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:05,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694005906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:05,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694005907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:05,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694005908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:05,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694005909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:05,950 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/b03b201dc1b34eb09a0de3efd6405581 2023-04-28 14:59:05,953 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/679c24c1d188453690150feef68da748 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/679c24c1d188453690150feef68da748 2023-04-28 14:59:05,957 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7fd2108ac9c35550edc28d90fafe4714/C of 7fd2108ac9c35550edc28d90fafe4714 into 679c24c1d188453690150feef68da748(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:05,957 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:05,957 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714., storeName=7fd2108ac9c35550edc28d90fafe4714/C, priority=13, startTime=1682693945460; duration=0sec 2023-04-28 14:59:05,957 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:06,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:06,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694006210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:06,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694006210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:06,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694006211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:06,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694006211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:06,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694006212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,358 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/2195513540f044eeae5cdf4126852a62 2023-04-28 14:59:06,365 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/ad2dfd65c80e4f1795202b55afd3a440 2023-04-28 14:59:06,369 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/b03b201dc1b34eb09a0de3efd6405581 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/b03b201dc1b34eb09a0de3efd6405581 2023-04-28 14:59:06,371 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/b03b201dc1b34eb09a0de3efd6405581, entries=150, sequenceid=67, filesize=11.6 K 2023-04-28 14:59:06,372 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/2195513540f044eeae5cdf4126852a62 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/2195513540f044eeae5cdf4126852a62 2023-04-28 14:59:06,374 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/2195513540f044eeae5cdf4126852a62, entries=150, sequenceid=67, filesize=11.6 K 2023-04-28 14:59:06,375 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/ad2dfd65c80e4f1795202b55afd3a440 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/ad2dfd65c80e4f1795202b55afd3a440 2023-04-28 14:59:06,378 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/ad2dfd65c80e4f1795202b55afd3a440, entries=150, sequenceid=67, filesize=11.6 K 2023-04-28 14:59:06,378 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 7fd2108ac9c35550edc28d90fafe4714 in 843ms, sequenceid=67, compaction requested=false 2023-04-28 14:59:06,378 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:06,378 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:06,378 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:06,378 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:06,378 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,378 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:06,379 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:06,379 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,379 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:06,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:06,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:06,380 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:06,380 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:06,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:06,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:06,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:06,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:06,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:06,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17131574[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:06,381 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:06,381 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:06,381 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:06,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:06,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:06,381 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:06,381 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:06,381 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:06,381 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,381 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:06,382 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,382 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:06,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:06,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:06,382 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:06,382 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:06,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:06,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:06,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:06,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:06,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:06,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:06,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:06,385 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:06,385 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:06,385 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:06,385 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:06,385 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:06,386 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:06,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:06,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:06,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:06,386 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:06,386 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:06,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:06,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:06,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:06,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:06,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:06,387 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:06,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:06,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:06,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:06,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,388 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:06,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:06,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,390 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:06,390 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,390 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:06,390 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:06,390 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:06,390 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:06,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:06,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:06,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:06,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:06,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:06,390 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,391 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:06,391 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:06,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:06,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:06,391 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:06,391 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:06,391 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:06,491 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:06,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:06,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:06,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:06,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:06,494 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:06,494 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:06,494 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:06,494 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:06,494 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:06,494 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:06,495 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,495 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:06,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:06,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:06,495 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,495 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:06,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:06,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:06,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:06,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:06,496 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:06,496 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:06,496 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:06,496 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:06,496 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:06,496 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:06,496 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:06,496 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:06,496 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:06,496 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:06,497 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7fd2108ac9c35550edc28d90fafe4714 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-28 14:59:06,497 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=A 2023-04-28 14:59:06,497 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:06,497 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=B 2023-04-28 14:59:06,497 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:06,497 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=C 2023-04-28 14:59:06,497 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:06,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:06,720 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:06,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:06,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694006729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:06,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694006730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:06,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694006733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,737 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:06,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694006733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:06,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694006734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:06,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694006835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,840 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:06,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694006836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,840 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:06,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694006837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,841 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:06,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694006838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,841 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:06,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694006838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:06,904 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/37e761bfd59d480eaf40a4350f34914d 2023-04-28 14:59:07,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:07,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694007038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,043 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:07,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694007040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,044 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:07,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694007041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,045 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:07,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694007042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,047 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:07,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694007043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,312 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/8849daae8a814f2eb9d8ed2b9dd2286c 2023-04-28 14:59:07,347 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:07,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694007344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:07,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694007344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:07,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694007345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:07,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694007346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,357 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:07,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694007349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,720 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/0612e02eb73344edbc4422dacca62f25 2023-04-28 14:59:07,723 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/37e761bfd59d480eaf40a4350f34914d as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/37e761bfd59d480eaf40a4350f34914d 2023-04-28 14:59:07,725 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/37e761bfd59d480eaf40a4350f34914d, entries=150, sequenceid=81, filesize=11.6 K 2023-04-28 14:59:07,726 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/8849daae8a814f2eb9d8ed2b9dd2286c as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/8849daae8a814f2eb9d8ed2b9dd2286c 2023-04-28 14:59:07,728 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/8849daae8a814f2eb9d8ed2b9dd2286c, entries=150, sequenceid=81, filesize=11.6 K 2023-04-28 14:59:07,729 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/0612e02eb73344edbc4422dacca62f25 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/0612e02eb73344edbc4422dacca62f25 2023-04-28 14:59:07,731 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/0612e02eb73344edbc4422dacca62f25, entries=150, sequenceid=81, filesize=11.6 K 2023-04-28 14:59:07,732 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 7fd2108ac9c35550edc28d90fafe4714 in 1236ms, sequenceid=81, compaction requested=true 2023-04-28 14:59:07,732 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:07,732 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:07,732 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:07,732 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:07,732 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,732 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:07,732 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:07,734 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,734 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:07,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:07,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:07,735 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:07,735 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:07,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:07,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:07,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:07,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:07,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:07,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72bff35c[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:07,736 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:07,736 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:07,736 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:07,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:07,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:07,736 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:07,736 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:07,736 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,736 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:07,736 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:07,737 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:07,737 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,737 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:07,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:07,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:07,737 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:07,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:07,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:07,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:07,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:07,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:07,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:07,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:07,739 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:07,739 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:07,739 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:07,739 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:07,739 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:07,740 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:07,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:07,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:07,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:07,740 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:07,740 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:07,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:07,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:07,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:07,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:07,740 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:07,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:07,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:07,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:07,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:07,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,741 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:07,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:07,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,742 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,742 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:07,742 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:07,742 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,742 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:07,742 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:07,742 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:07,742 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:07,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:07,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:07,742 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:07,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:07,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:07,742 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:07,743 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:07,743 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:07,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:07,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:07,843 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:07,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:07,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:07,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:07,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:07,846 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:07,846 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:07,846 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:07,846 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:07,846 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:07,846 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:07,847 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:07,847 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:07,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:07,847 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,847 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:07,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:07,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:07,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:07,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:07,847 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:07,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:07,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:07,848 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:07,848 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:07,848 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:07,848 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:07,848 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:07,848 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:07,848 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:07,848 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7fd2108ac9c35550edc28d90fafe4714 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-28 14:59:07,848 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=A 2023-04-28 14:59:07,848 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:07,848 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=B 2023-04-28 14:59:07,849 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:07,849 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=C 2023-04-28 14:59:07,849 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:07,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:07,854 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:07,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:07,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694007862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:07,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694007864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:07,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694007885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:07,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694007885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:07,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694007885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:07,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694007986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:07,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:07,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694007986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:08,002 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:08,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694007997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:08,002 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:08,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694007997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:08,002 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:08,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694007998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:08,196 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:08,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694008191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:08,197 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:08,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694008192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:08,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:08,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694008204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:08,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:08,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694008204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:08,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:08,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694008204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:08,256 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/5a9596747a664d78852391077d56fe98 2023-04-28 14:59:08,263 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/a76499e561724911af5e02445b5a3348 2023-04-28 14:59:08,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:08,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694008497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:08,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:08,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694008498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:08,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:08,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694008508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:08,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:08,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694008508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:08,515 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:08,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694008510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:08,671 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/144b8d8261864e3280222948f3e20a89 2023-04-28 14:59:08,675 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/5a9596747a664d78852391077d56fe98 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/5a9596747a664d78852391077d56fe98 2023-04-28 14:59:08,677 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/5a9596747a664d78852391077d56fe98, entries=150, sequenceid=100, filesize=11.6 K 2023-04-28 14:59:08,678 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/a76499e561724911af5e02445b5a3348 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/a76499e561724911af5e02445b5a3348 2023-04-28 14:59:08,680 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/a76499e561724911af5e02445b5a3348, entries=150, sequenceid=100, filesize=11.6 K 2023-04-28 14:59:08,681 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/144b8d8261864e3280222948f3e20a89 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/144b8d8261864e3280222948f3e20a89 2023-04-28 14:59:08,683 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/144b8d8261864e3280222948f3e20a89, entries=150, sequenceid=100, filesize=11.6 K 2023-04-28 14:59:08,684 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 7fd2108ac9c35550edc28d90fafe4714 in 836ms, sequenceid=100, compaction requested=true 2023-04-28 14:59:08,684 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:08,684 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:08,684 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:08,684 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:08,684 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:08,684 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:08,684 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:08,685 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:08,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:08,685 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:08,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:08,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:08,685 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:08,685 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:08,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:08,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:08,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:08,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:08,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:08,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:08,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49e2a478[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:08,686 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:08,687 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:08,687 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:08,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:08,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:08,687 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:08,687 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:08,687 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:08,687 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:08,687 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:08,688 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:08,688 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:08,688 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:08,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:08,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:08,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:08,688 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:08,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:08,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:08,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:08,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:08,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:08,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:08,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:08,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:08,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:08,690 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:08,690 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:08,690 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:08,690 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:08,690 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:08,691 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:08,691 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:08,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:08,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:08,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:08,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:08,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:08,691 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:08,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:08,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:08,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:08,691 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:08,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:08,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:08,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:08,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:08,692 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:08,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:08,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:08,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:08,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:08,693 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:08,693 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:08,693 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:08,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:08,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:08,693 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:08,693 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:08,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:08,693 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:08,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:08,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:08,693 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:08,694 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:08,694 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:08,694 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:08,694 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:08,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:08,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:08,694 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:08,794 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:08,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:08,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:08,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:08,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:08,797 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:08,797 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:08,797 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:08,797 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:08,798 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:08,798 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:08,798 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:08,798 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:08,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:08,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:08,799 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:08,799 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:08,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:08,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:08,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:08,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:08,799 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:08,799 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:08,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:08,800 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:08,800 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:08,800 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:08,800 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:08,800 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:08,800 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:08,800 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:08,800 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7fd2108ac9c35550edc28d90fafe4714 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-28 14:59:08,800 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=A 2023-04-28 14:59:08,800 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:08,800 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=B 2023-04-28 14:59:08,800 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:08,800 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=C 2023-04-28 14:59:08,800 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:09,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:09,006 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:09,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:09,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694009025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:09,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:09,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694009026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:09,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:09,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694009030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:09,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:09,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694009031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:09,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:09,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694009032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:09,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:09,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694009133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:09,142 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:09,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694009137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:09,142 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:09,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694009138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:09,142 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:09,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694009139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:09,142 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:09,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694009139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:09,211 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/81a86d93e59c4910b77b25f268bbc5b2 2023-04-28 14:59:09,339 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:09,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694009336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:09,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:09,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694009343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:09,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:09,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694009343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:09,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:09,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694009344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:09,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:09,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694009344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:09,620 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/6c316ec39aee4671bc3185a7bab1e54e 2023-04-28 14:59:09,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:09,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694009640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:09,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:09,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694009647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:09,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:09,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694009650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:09,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:09,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694009651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:09,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:09,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694009651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,029 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/46978d3cb8e94a778a67b341d677f992 2023-04-28 14:59:10,032 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/81a86d93e59c4910b77b25f268bbc5b2 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/81a86d93e59c4910b77b25f268bbc5b2 2023-04-28 14:59:10,035 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/81a86d93e59c4910b77b25f268bbc5b2, entries=150, sequenceid=113, filesize=11.6 K 2023-04-28 14:59:10,036 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/6c316ec39aee4671bc3185a7bab1e54e as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/6c316ec39aee4671bc3185a7bab1e54e 2023-04-28 14:59:10,038 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/6c316ec39aee4671bc3185a7bab1e54e, entries=150, sequenceid=113, filesize=11.6 K 2023-04-28 14:59:10,040 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/46978d3cb8e94a778a67b341d677f992 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/46978d3cb8e94a778a67b341d677f992 2023-04-28 14:59:10,042 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/46978d3cb8e94a778a67b341d677f992, entries=150, sequenceid=113, filesize=11.6 K 2023-04-28 14:59:10,043 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 7fd2108ac9c35550edc28d90fafe4714 in 1243ms, sequenceid=113, compaction requested=true 2023-04-28 14:59:10,043 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:10,043 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:10,043 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:10,043 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:10,043 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,043 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:10,043 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:10,044 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,044 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:10,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:10,044 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,044 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:10,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:10,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:10,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:10,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:10,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6263c6f8[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:10,045 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:10,046 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,046 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,046 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:10,046 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:10,046 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,046 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:10,046 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:10,047 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:10,047 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:10,047 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:10,048 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:10,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:10,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:10,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:10,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:10,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:10,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:10,050 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:10,050 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:10,050 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:10,050 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:10,050 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:10,052 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,052 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:10,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:10,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,052 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:10,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:10,052 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:10,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:10,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:10,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,053 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:10,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,054 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,054 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:10,054 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:10,054 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,055 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,055 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,055 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:10,054 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:10,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:10,055 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:10,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:10,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:10,055 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,055 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:10,055 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:10,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:10,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:10,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:10,152 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7fd2108ac9c35550edc28d90fafe4714 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-28 14:59:10,152 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=A 2023-04-28 14:59:10,152 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:10,152 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=B 2023-04-28 14:59:10,152 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:10,152 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=C 2023-04-28 14:59:10,152 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:10,155 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:10,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:10,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:10,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:10,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:10,164 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:10,164 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:10,164 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:10,165 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:10,165 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,165 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,165 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,165 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:10,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:10,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:10,166 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,166 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:10,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:10,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:10,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:10,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:10,176 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:10,176 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:10,177 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:10,178 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:10,178 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:10,178 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:10,178 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:10,178 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:10,178 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:10,178 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:10,178 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:10,178 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,178 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:10,178 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:10,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:10,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694010168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,179 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,179 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:10,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:10,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:10,179 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,179 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:10,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:10,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,180 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:10,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:10,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694010169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,180 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:10,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694010170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:10,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dd4d775[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:10,181 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:10,181 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,182 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,182 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:10,182 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:10,182 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:10,182 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,182 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:10,183 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,183 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:10,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:10,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:10,183 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:10,183 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:10,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:10,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:10,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:10,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:10,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:10,185 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:10,185 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:10,185 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:10,185 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:10,185 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:10,186 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,186 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:10,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:10,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,186 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:10,186 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:10,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:10,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:10,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:10,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694010178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:10,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694010179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:10,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,193 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:10,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,194 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,194 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:10,194 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:10,194 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:10,194 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:10,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:10,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:10,195 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,195 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,195 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:10,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:10,195 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:10,196 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,196 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:10,196 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:10,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:10,283 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:10,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694010280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,283 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:10,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694010281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,283 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:10,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694010281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:10,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694010289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:10,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694010289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,296 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:10,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:10,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:10,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:10,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:10,298 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:10,299 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:10,299 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:10,299 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:10,299 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,299 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,299 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:10,300 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:10,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:10,300 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,300 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:10,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:10,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,300 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:10,301 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:10,301 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:10,301 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:10,301 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:10,301 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:10,301 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:10,301 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:10,301 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:10,301 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:10,301 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:10,301 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:10,301 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:10,301 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:10,301 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,301 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:10,301 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:10,302 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,302 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:10,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:10,302 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,302 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:10,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:10,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:10,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:10,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:10,303 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:10,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60fcfc80[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:10,304 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,304 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,304 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:10,304 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:10,304 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:10,304 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,304 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:10,305 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,305 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:10,305 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:10,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:10,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:10,305 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:10,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:10,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:10,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:10,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:10,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:10,307 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:10,307 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:10,307 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:10,307 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:10,307 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:10,308 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,308 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:10,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:10,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,308 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:10,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:10,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:10,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:10,308 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:10,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,309 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:10,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,310 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,311 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:10,311 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,311 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:10,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:10,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:10,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:10,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:10,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:10,311 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:10,311 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,311 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:10,311 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,311 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:10,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:10,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:10,311 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,311 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,311 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,411 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:10,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:10,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:10,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:10,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:10,414 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:10,414 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:10,414 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:10,414 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:10,414 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,415 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,415 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:10,415 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:10,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:10,415 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,415 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:10,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:10,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:10,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:10,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:10,416 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:10,416 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:10,416 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:10,416 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:10,416 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:10,416 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:10,416 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:10,417 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:10,417 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:10,417 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:10,417 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:10,417 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,417 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:10,417 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:10,417 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,417 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:10,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:10,418 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,418 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:10,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:10,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:10,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:10,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:10,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c614583[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:10,419 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:10,419 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,419 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,419 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:10,419 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:10,419 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,420 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:10,419 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:10,420 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:10,420 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:10,420 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,420 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:10,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:10,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:10,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:10,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:10,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:10,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:10,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:10,422 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:10,422 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:10,422 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:10,422 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:10,422 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:10,423 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,423 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:10,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:10,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,423 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:10,423 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:10,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:10,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:10,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:10,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,424 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:10,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,426 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,426 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,426 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,426 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,426 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:10,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:10,426 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:10,426 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:10,426 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,426 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,426 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:10,426 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:10,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:10,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:10,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:10,426 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:10,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:10,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:10,486 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:10,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694010484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,487 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:10,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694010484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,487 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:10,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694010485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,498 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:10,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694010494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,498 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:10,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694010494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,526 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:10,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:10,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:10,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:10,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:10,529 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:10,529 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:10,529 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:10,529 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:10,529 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,529 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,530 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:10,530 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:10,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:10,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,530 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,530 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:10,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:10,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,531 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:10,531 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:10,531 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:10,531 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:10,531 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:10,531 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:10,531 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:10,531 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:10,531 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:10,531 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:10,531 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:10,531 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:10,531 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:10,531 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:10,531 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,531 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:10,531 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:10,532 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,532 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,532 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:10,533 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:10,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:10,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:10,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:10,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:10,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:10,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40c900f1[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:10,534 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:10,534 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,535 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,535 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:10,535 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:10,535 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,535 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:10,535 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:10,536 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,536 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:10,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:10,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:10,536 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:10,536 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:10,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:10,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:10,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:10,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:10,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:10,538 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:10,538 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:10,538 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:10,538 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:10,538 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:10,539 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,539 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:10,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:10,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,539 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:10,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:10,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:10,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:10,539 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:10,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,541 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:10,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,542 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,542 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,542 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:10,542 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:10,542 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:10,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:10,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:10,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:10,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:10,542 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,543 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,543 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:10,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:10,543 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,543 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:10,543 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,543 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:10,575 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/4e8d1b80884b464f81edb7b632c0d0ad 2023-04-28 14:59:10,643 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:10,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:10,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:10,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:10,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:10,646 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:10,646 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:10,646 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:10,646 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:10,646 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,646 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,647 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:10,647 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:10,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:10,647 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,647 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:10,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:10,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,648 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:10,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:10,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:10,648 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:10,648 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:10,648 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:10,648 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:10,648 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:10,648 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:10,648 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:10,648 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:10,648 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:10,648 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:10,649 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:10,649 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,649 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:10,649 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:10,649 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,649 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:10,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:10,649 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,650 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:10,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:10,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:10,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:10,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:10,651 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:10,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bed18e2[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:10,651 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,651 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,651 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:10,651 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:10,651 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:10,651 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,651 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:10,652 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,652 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:10,652 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:10,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:10,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:10,652 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:10,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:10,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:10,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:10,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:10,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:10,654 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:10,654 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:10,654 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:10,654 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:10,654 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:10,655 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,655 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:10,655 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:10,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:10,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:10,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:10,655 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:10,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:10,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,656 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:10,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,658 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,658 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,658 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:10,658 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:10,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:10,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:10,658 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,658 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:10,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:10,658 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,658 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:10,658 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:10,658 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,658 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:10,658 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:10,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:10,758 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:10,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:10,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:10,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:10,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:10,761 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:10,761 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:10,761 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:10,761 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:10,762 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,762 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,763 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,763 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:10,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:10,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:10,764 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,764 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:10,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:10,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:10,766 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:10,766 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:10,766 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:10,766 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:10,766 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:10,766 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:10,766 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:10,766 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:10,766 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:10,766 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:10,766 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:10,766 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:10,766 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,767 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:10,767 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:10,767 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,767 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:10,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:10,768 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,768 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:10,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:10,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:10,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:10,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:10,769 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:10,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cf2fde3[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:10,769 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,770 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,770 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,770 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:10,770 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:10,770 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:10,770 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:10,771 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,771 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:10,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:10,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:10,771 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:10,771 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:10,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:10,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:10,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:10,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:10,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:10,774 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:10,774 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:10,774 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:10,774 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:10,774 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:10,775 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,775 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:10,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:10,775 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:10,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:10,776 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:10,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:10,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:10,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,777 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:10,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,780 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:10,780 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:10,780 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,780 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,780 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,780 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,780 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,780 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:10,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:10,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:10,780 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:10,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:10,780 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:10,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:10,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:10,781 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:10,781 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:10,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:10,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694010788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:10,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:10,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694010790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694010790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,806 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:10,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694010799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,806 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:10,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694010800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,881 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:10,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:10,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:10,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:10,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:10,883 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:10,883 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:10,883 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:10,883 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:10,884 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,884 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,884 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:10,884 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:10,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:10,885 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,885 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:10,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:10,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,885 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:10,885 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:10,886 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:10,886 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:10,886 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:10,886 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:10,886 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:10,886 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:10,886 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:10,886 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:10,886 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:10,886 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:10,886 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:10,886 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:10,886 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,886 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:10,886 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:10,887 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:10,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:10,888 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:10,888 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,888 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:10,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:10,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:10,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:10,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a09eca3[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:10,889 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:10,889 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,889 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,890 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:10,890 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:10,890 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:10,890 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,890 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:10,890 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,890 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:10,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:10,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:10,890 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:10,890 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:10,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:10,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:10,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:10,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:10,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:10,892 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:10,892 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:10,892 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:10,892 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:10,892 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:10,893 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,893 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:10,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:10,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,893 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:10,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:10,893 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:10,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:10,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:10,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,894 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:10,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:10,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,898 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:10,898 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:10,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:10,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:10,898 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,898 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:10,898 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:10,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:10,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:10,898 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:10,898 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:10,898 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:10,899 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:10,899 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:10,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:10,899 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:10,899 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:10,983 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/b608dd33a4064230b2c6cab492c6212d 2023-04-28 14:59:10,999 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:10,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:11,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:11,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:11,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:11,001 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:11,001 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:11,001 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:11,001 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:11,002 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,002 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:11,002 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:11,002 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:11,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:11,003 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,003 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:11,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:11,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:11,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:11,003 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:11,003 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:11,003 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:11,004 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:11,004 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:11,004 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:11,004 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:11,004 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:11,004 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:11,004 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:11,004 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:11,004 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:11,004 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:11,004 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:11,004 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,004 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:11,004 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:11,005 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,005 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:11,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:11,005 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,005 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:11,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:11,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:11,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:11,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:11,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:11,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32cfbd9b[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:11,006 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:11,006 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,006 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,007 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,007 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:11,007 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:11,007 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:11,007 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:11,007 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,007 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:11,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:11,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:11,007 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:11,008 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:11,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:11,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:11,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:11,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:11,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:11,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:11,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:11,009 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:11,009 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:11,009 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:11,009 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:11,009 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:11,010 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,010 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:11,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:11,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,010 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:11,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:11,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:11,011 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:11,011 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:11,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:11,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,013 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,013 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:11,013 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:11,013 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:11,013 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:11,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:11,013 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,013 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:11,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:11,013 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,013 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:11,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:11,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:11,013 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,013 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:11,013 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:11,113 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:11,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:11,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:11,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:11,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:11,116 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:11,116 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:11,116 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:11,116 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:11,116 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,117 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:11,117 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,117 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:11,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:11,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:11,117 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,117 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:11,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:11,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:11,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:11,118 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:11,118 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:11,118 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:11,118 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:11,118 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:11,118 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:11,119 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:11,119 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:11,119 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:11,119 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:11,119 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:11,119 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:11,119 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:11,119 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:11,119 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,119 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:11,119 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:11,120 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,120 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:11,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:11,120 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,120 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:11,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:11,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:11,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:11,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:11,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:11,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51868928[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:11,121 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:11,121 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,122 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,122 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:11,122 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:11,122 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:11,122 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,123 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:11,123 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,123 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:11,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:11,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:11,123 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:11,123 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:11,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:11,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:11,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:11,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:11,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:11,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:11,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:11,125 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:11,125 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:11,125 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:11,125 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:11,125 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:11,126 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,126 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,126 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:11,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:11,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:11,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,126 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:11,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:11,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:11,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:11,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:11,127 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:11,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:11,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:11,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,128 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,128 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:11,128 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:11,128 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:11,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:11,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:11,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:11,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:11,128 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:11,128 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,128 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:11,129 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:11,129 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:11,129 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,129 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:11,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:11,129 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:11,229 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:11,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:11,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:11,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:11,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:11,232 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:11,232 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:11,232 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:11,232 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:11,232 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,232 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:11,233 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:11,233 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:11,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:11,233 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,233 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:11,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:11,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:11,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:11,234 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:11,234 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:11,234 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:11,234 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:11,234 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:11,234 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:11,234 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:11,234 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:11,234 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:11,234 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:11,234 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:11,234 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:11,234 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:11,234 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:11,234 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,234 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:11,234 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:11,235 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,235 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:11,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:11,235 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:11,235 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:11,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:11,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:11,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:11,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:11,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a006b1e[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:11,236 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:11,236 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,237 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,237 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:11,237 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,237 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:11,237 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:11,237 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:11,238 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,238 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:11,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:11,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:11,238 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:11,238 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:11,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:11,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:11,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:11,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:11,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:11,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:11,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:11,239 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:11,239 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:11,239 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:11,240 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:11,240 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:11,240 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,240 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:11,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:11,240 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:11,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:11,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:11,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:11,241 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:11,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:11,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:11,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,242 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:11,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:11,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,244 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,244 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:11,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:11,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:11,244 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:11,244 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:11,244 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:11,244 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:11,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:11,244 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:11,244 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:11,244 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,244 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:11,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:11,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:11,244 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,244 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,297 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:11,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694011293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,306 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:11,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694011301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,309 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:11,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694011302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,318 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:11,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694011314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,318 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:11,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694011314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,344 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:11,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:11,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:11,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:11,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:11,347 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:11,347 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:11,347 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:11,347 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:11,348 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,348 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:11,348 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,348 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:11,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:11,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:11,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:11,349 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,349 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:11,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:11,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:11,349 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:11,349 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:11,349 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:11,349 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:11,350 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:11,350 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:11,350 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:11,350 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:11,350 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:11,350 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:11,350 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:11,350 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:11,350 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:11,350 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:11,350 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,350 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:11,350 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:11,351 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,351 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:11,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:11,351 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,351 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:11,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:11,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:11,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:11,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:11,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:11,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f99fa09[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:11,354 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:11,354 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,355 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,355 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:11,355 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:11,355 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:11,355 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,355 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:11,355 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,355 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:11,355 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:11,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:11,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:11,355 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:11,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:11,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:11,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:11,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:11,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:11,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:11,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:11,357 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:11,357 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:11,357 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:11,357 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:11,357 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:11,358 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,358 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:11,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:11,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,358 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:11,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:11,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:11,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:11,359 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:11,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:11,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:11,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,359 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:11,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:11,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,361 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,361 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:11,361 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:11,361 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,361 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,361 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:11,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:11,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:11,361 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:11,361 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:11,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:11,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:11,361 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:11,361 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:11,361 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:11,361 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:11,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:11,391 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/37fb6eef6f544122a308ec23dcbfb9e6 2023-04-28 14:59:11,395 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/4e8d1b80884b464f81edb7b632c0d0ad as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/4e8d1b80884b464f81edb7b632c0d0ad 2023-04-28 14:59:11,398 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/4e8d1b80884b464f81edb7b632c0d0ad, entries=200, sequenceid=133, filesize=14.0 K 2023-04-28 14:59:11,398 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/b608dd33a4064230b2c6cab492c6212d as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/b608dd33a4064230b2c6cab492c6212d 2023-04-28 14:59:11,401 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/b608dd33a4064230b2c6cab492c6212d, entries=150, sequenceid=133, filesize=11.7 K 2023-04-28 14:59:11,402 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/37fb6eef6f544122a308ec23dcbfb9e6 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/37fb6eef6f544122a308ec23dcbfb9e6 2023-04-28 14:59:11,410 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/37fb6eef6f544122a308ec23dcbfb9e6, entries=150, sequenceid=133, filesize=11.7 K 2023-04-28 14:59:11,411 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 7fd2108ac9c35550edc28d90fafe4714 in 1259ms, sequenceid=133, compaction requested=true 2023-04-28 14:59:11,411 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:11,411 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:11,411 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-28 14:59:11,411 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-28 14:59:11,411 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:11,411 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:11,413 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73909 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-28 14:59:11,413 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71519 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-28 14:59:11,413 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): 7fd2108ac9c35550edc28d90fafe4714/A is initiating minor compaction (all files) 2023-04-28 14:59:11,413 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 7fd2108ac9c35550edc28d90fafe4714/B is initiating minor compaction (all files) 2023-04-28 14:59:11,413 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7fd2108ac9c35550edc28d90fafe4714/A in TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:11,413 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7fd2108ac9c35550edc28d90fafe4714/B in TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:11,413 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/a6a105000c874b78b6788e28ae1ad3c7, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/2195513540f044eeae5cdf4126852a62, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/8849daae8a814f2eb9d8ed2b9dd2286c, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/a76499e561724911af5e02445b5a3348, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/6c316ec39aee4671bc3185a7bab1e54e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/b608dd33a4064230b2c6cab492c6212d] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp, totalSize=69.8 K 2023-04-28 14:59:11,413 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/3109db1b49e3434497771ef287f1faa9, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/b03b201dc1b34eb09a0de3efd6405581, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/37e761bfd59d480eaf40a4350f34914d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/5a9596747a664d78852391077d56fe98, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/81a86d93e59c4910b77b25f268bbc5b2, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/4e8d1b80884b464f81edb7b632c0d0ad] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp, totalSize=72.2 K 2023-04-28 14:59:11,413 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting a6a105000c874b78b6788e28ae1ad3c7, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1682693943801 2023-04-28 14:59:11,413 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 3109db1b49e3434497771ef287f1faa9, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1682693943801 2023-04-28 14:59:11,413 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 2195513540f044eeae5cdf4126852a62, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1682693944963 2023-04-28 14:59:11,413 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting b03b201dc1b34eb09a0de3efd6405581, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1682693944963 2023-04-28 14:59:11,414 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 8849daae8a814f2eb9d8ed2b9dd2286c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1682693945584 2023-04-28 14:59:11,414 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 37e761bfd59d480eaf40a4350f34914d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1682693945584 2023-04-28 14:59:11,414 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting a76499e561724911af5e02445b5a3348, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1682693946732 2023-04-28 14:59:11,414 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 5a9596747a664d78852391077d56fe98, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1682693946732 2023-04-28 14:59:11,414 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 6c316ec39aee4671bc3185a7bab1e54e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1682693947860 2023-04-28 14:59:11,414 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 81a86d93e59c4910b77b25f268bbc5b2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1682693947860 2023-04-28 14:59:11,414 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting b608dd33a4064230b2c6cab492c6212d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1682693949024 2023-04-28 14:59:11,414 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 4e8d1b80884b464f81edb7b632c0d0ad, keycount=200, bloomtype=ROW, size=14.0 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1682693949024 2023-04-28 14:59:11,433 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7fd2108ac9c35550edc28d90fafe4714#A#compaction#381 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:59:11,434 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 7fd2108ac9c35550edc28d90fafe4714#B#compaction#382 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:59:11,445 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/29779d727fa94e2fa235dc344e25fdb2 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/29779d727fa94e2fa235dc344e25fdb2 2023-04-28 14:59:11,449 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 7fd2108ac9c35550edc28d90fafe4714/B of 7fd2108ac9c35550edc28d90fafe4714 into 29779d727fa94e2fa235dc344e25fdb2(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:11,449 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:11,449 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714., storeName=7fd2108ac9c35550edc28d90fafe4714/B, priority=10, startTime=1682693951411; duration=0sec 2023-04-28 14:59:11,449 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:11,449 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-28 14:59:11,451 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71519 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-28 14:59:11,451 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 7fd2108ac9c35550edc28d90fafe4714/C is initiating minor compaction (all files) 2023-04-28 14:59:11,451 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7fd2108ac9c35550edc28d90fafe4714/C in TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:11,451 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/679c24c1d188453690150feef68da748, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/ad2dfd65c80e4f1795202b55afd3a440, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/0612e02eb73344edbc4422dacca62f25, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/144b8d8261864e3280222948f3e20a89, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/46978d3cb8e94a778a67b341d677f992, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/37fb6eef6f544122a308ec23dcbfb9e6] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp, totalSize=69.8 K 2023-04-28 14:59:11,451 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 679c24c1d188453690150feef68da748, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1682693943801 2023-04-28 14:59:11,452 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting ad2dfd65c80e4f1795202b55afd3a440, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1682693944963 2023-04-28 14:59:11,452 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 0612e02eb73344edbc4422dacca62f25, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1682693945584 2023-04-28 14:59:11,452 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 144b8d8261864e3280222948f3e20a89, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1682693946732 2023-04-28 14:59:11,452 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 46978d3cb8e94a778a67b341d677f992, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1682693947860 2023-04-28 14:59:11,453 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 37fb6eef6f544122a308ec23dcbfb9e6, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1682693949024 2023-04-28 14:59:11,454 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/cb686a39049547249bfcc2530a797c6b as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/cb686a39049547249bfcc2530a797c6b 2023-04-28 14:59:11,462 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:11,462 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 7fd2108ac9c35550edc28d90fafe4714/A of 7fd2108ac9c35550edc28d90fafe4714 into cb686a39049547249bfcc2530a797c6b(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:11,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:11,462 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:11,462 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714., storeName=7fd2108ac9c35550edc28d90fafe4714/A, priority=10, startTime=1682693951411; duration=0sec 2023-04-28 14:59:11,462 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:11,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:11,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:11,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:11,464 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:11,464 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:11,464 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:11,464 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:11,465 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,465 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:11,465 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:11,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:11,465 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 7fd2108ac9c35550edc28d90fafe4714#C#compaction#383 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:59:11,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:11,466 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:11,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:11,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:11,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:11,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:11,468 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:11,468 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:11,470 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:11,470 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:11,470 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:11,470 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:11,470 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:11,472 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:11,472 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:11,472 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:11,472 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:11,472 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7fd2108ac9c35550edc28d90fafe4714 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-28 14:59:11,472 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=A 2023-04-28 14:59:11,472 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:11,472 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=B 2023-04-28 14:59:11,472 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:11,472 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=C 2023-04-28 14:59:11,472 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:11,883 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/e9ffd2c947bf43cbae8231b08c845aaf as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/e9ffd2c947bf43cbae8231b08c845aaf 2023-04-28 14:59:11,886 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 7fd2108ac9c35550edc28d90fafe4714/C of 7fd2108ac9c35550edc28d90fafe4714 into e9ffd2c947bf43cbae8231b08c845aaf(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:11,886 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:11,886 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714., storeName=7fd2108ac9c35550edc28d90fafe4714/C, priority=10, startTime=1682693951411; duration=0sec 2023-04-28 14:59:11,886 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:11,892 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/c09d8f4661ba460298d7cb221cd4eb5c 2023-04-28 14:59:12,299 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/e02a9df85ab84487a8e12b6d7418bc94 2023-04-28 14:59:12,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:12,302 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:12,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:12,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694012324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:12,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694012325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:12,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694012326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:12,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694012327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:12,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694012331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:12,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694012433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:12,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694012433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:12,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694012433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:12,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694012434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:12,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694012437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:12,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694012636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,640 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:12,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694012637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,640 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:12,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694012637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,640 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:12,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694012638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:12,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694012641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,716 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/7b3dc46c66394f26b1ceb860b6d14cbe 2023-04-28 14:59:12,719 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/c09d8f4661ba460298d7cb221cd4eb5c as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/c09d8f4661ba460298d7cb221cd4eb5c 2023-04-28 14:59:12,722 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/c09d8f4661ba460298d7cb221cd4eb5c, entries=150, sequenceid=147, filesize=11.8 K 2023-04-28 14:59:12,722 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/e02a9df85ab84487a8e12b6d7418bc94 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/e02a9df85ab84487a8e12b6d7418bc94 2023-04-28 14:59:12,725 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/e02a9df85ab84487a8e12b6d7418bc94, entries=150, sequenceid=147, filesize=11.8 K 2023-04-28 14:59:12,726 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/7b3dc46c66394f26b1ceb860b6d14cbe as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/7b3dc46c66394f26b1ceb860b6d14cbe 2023-04-28 14:59:12,739 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/7b3dc46c66394f26b1ceb860b6d14cbe, entries=150, sequenceid=147, filesize=11.8 K 2023-04-28 14:59:12,739 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 7fd2108ac9c35550edc28d90fafe4714 in 1267ms, sequenceid=147, compaction requested=false 2023-04-28 14:59:12,739 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:12,739 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:12,739 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:12,739 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:12,740 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,740 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:12,740 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:12,740 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,740 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:12,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:12,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:12,741 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:12,741 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:12,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:12,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:12,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:12,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:12,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:12,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7075ab04[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:12,742 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:12,742 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:12,743 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:12,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:12,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:12,743 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:12,743 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:12,743 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,743 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:12,743 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:12,744 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:12,744 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:12,744 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:12,744 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:12,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:12,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:12,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:12,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:12,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:12,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:12,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:12,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:12,745 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:12,745 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:12,745 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:12,746 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:12,746 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:12,746 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:12,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:12,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:12,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:12,746 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:12,747 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:12,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:12,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:12,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:12,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:12,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:12,747 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:12,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:12,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:12,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:12,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,748 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:12,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:12,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,750 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,750 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:12,750 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:12,750 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:12,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:12,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:12,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:12,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:12,750 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:12,750 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:12,750 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:12,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:12,750 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:12,750 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:12,750 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:12,750 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:12,750 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:12,850 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:12,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:12,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:12,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:12,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:12,854 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:12,854 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:12,854 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:12,854 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:12,855 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:12,855 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:12,855 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,855 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:12,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:12,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:12,856 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,856 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:12,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:12,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:12,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:12,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:12,856 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:12,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:12,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:12,856 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:12,856 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:12,856 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:12,857 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:12,857 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:12,857 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:12,857 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:12,857 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7fd2108ac9c35550edc28d90fafe4714 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-28 14:59:12,857 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=A 2023-04-28 14:59:12,857 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:12,857 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=B 2023-04-28 14:59:12,857 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:12,857 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=C 2023-04-28 14:59:12,857 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:12,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:12,941 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:12,951 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:12,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694012948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,954 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:12,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694012949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,954 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:12,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694012949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,954 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:12,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694012951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:12,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:12,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694012952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:13,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:13,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694013052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:13,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:13,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694013055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:13,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:13,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694013055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:13,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:13,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694013055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:13,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:13,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694013057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:13,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:13,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694013257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:13,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:13,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694013257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:13,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:13,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694013257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:13,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:13,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694013258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:13,263 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/d6bb6029c1864285bb8fec687db20309 2023-04-28 14:59:13,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:13,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694013266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:13,563 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:13,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694013562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:13,563 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:13,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694013562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:13,568 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:13,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694013564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:13,568 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:13,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694013564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:13,573 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:13,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694013569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:13,671 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/e40bf8498de141f5bf6bd51aa1b866c7 2023-04-28 14:59:14,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:14,073 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:14,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694014067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:14,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694014068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:14,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:14,073 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:14,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694014068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:14,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694014068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:14,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:14,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694014077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:14,080 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/da83122cb707457aa128a7a0fa89de42 2023-04-28 14:59:14,085 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/d6bb6029c1864285bb8fec687db20309 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/d6bb6029c1864285bb8fec687db20309 2023-04-28 14:59:14,089 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/d6bb6029c1864285bb8fec687db20309, entries=150, sequenceid=168, filesize=11.8 K 2023-04-28 14:59:14,090 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/e40bf8498de141f5bf6bd51aa1b866c7 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/e40bf8498de141f5bf6bd51aa1b866c7 2023-04-28 14:59:14,094 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/e40bf8498de141f5bf6bd51aa1b866c7, entries=150, sequenceid=168, filesize=11.8 K 2023-04-28 14:59:14,094 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/da83122cb707457aa128a7a0fa89de42 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/da83122cb707457aa128a7a0fa89de42 2023-04-28 14:59:14,099 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/da83122cb707457aa128a7a0fa89de42, entries=150, sequenceid=168, filesize=11.8 K 2023-04-28 14:59:14,099 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 7fd2108ac9c35550edc28d90fafe4714 in 1242ms, sequenceid=168, compaction requested=true 2023-04-28 14:59:14,100 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:14,100 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:14,100 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:14,100 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:14,100 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:14,100 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:14,100 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:14,101 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:14,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:14,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:14,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:14,101 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:14,101 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:14,101 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:14,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:14,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:14,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:14,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:14,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:14,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:14,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@155fa3eb[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:14,104 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:14,104 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:14,104 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:14,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:14,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:14,104 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:14,104 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:14,104 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:14,104 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:14,105 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:14,105 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:14,105 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:14,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:14,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:14,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:14,105 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:14,105 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:14,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:14,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:14,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:14,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:14,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:14,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:14,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:14,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:14,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:14,109 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:14,109 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:14,109 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:14,109 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:14,109 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:14,110 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:14,110 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:14,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:14,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:14,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:14,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:14,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:14,110 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:14,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:14,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:14,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:14,111 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:14,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:14,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:14,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:14,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:14,113 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:14,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:14,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:14,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:14,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:14,116 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:14,116 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:14,116 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:14,116 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:14,116 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:14,116 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:14,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:14,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:14,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:14,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:14,116 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:14,116 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:14,117 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:14,117 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:14,117 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:14,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:14,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:14,117 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:14,117 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:14,217 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:14,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:14,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:14,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:14,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:14,220 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:14,220 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:14,220 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:14,220 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:14,220 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:14,220 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:14,222 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:14,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:14,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:14,222 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:14,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:14,222 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:14,222 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:14,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:14,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:14,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:14,223 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:14,223 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:14,223 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:14,223 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:14,223 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:14,223 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:14,224 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:14,224 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:14,224 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:14,224 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:14,224 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7fd2108ac9c35550edc28d90fafe4714 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-28 14:59:14,225 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=A 2023-04-28 14:59:14,225 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:14,225 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=B 2023-04-28 14:59:14,225 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:14,225 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=C 2023-04-28 14:59:14,225 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:14,636 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/28e6c89bfbec476eb9ae3e404394c0c8 2023-04-28 14:59:15,043 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/a0aa5ab2e85646a1ac7cd2a0828c3682 2023-04-28 14:59:15,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:15,079 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:15,106 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:15,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694015097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,106 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:15,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694015098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,108 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:15,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694015103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:15,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694015105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:15,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694015106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:15,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694015207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:15,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694015208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:15,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694015209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:15,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694015214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:15,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694015215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:15,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694015408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,417 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:15,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694015415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:15,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694015416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:15,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694015419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:15,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694015419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,450 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/383ce69044f049709627266e41ad430a 2023-04-28 14:59:15,453 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/28e6c89bfbec476eb9ae3e404394c0c8 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/28e6c89bfbec476eb9ae3e404394c0c8 2023-04-28 14:59:15,456 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/28e6c89bfbec476eb9ae3e404394c0c8, entries=150, sequenceid=180, filesize=11.8 K 2023-04-28 14:59:15,456 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/a0aa5ab2e85646a1ac7cd2a0828c3682 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/a0aa5ab2e85646a1ac7cd2a0828c3682 2023-04-28 14:59:15,459 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/a0aa5ab2e85646a1ac7cd2a0828c3682, entries=150, sequenceid=180, filesize=11.8 K 2023-04-28 14:59:15,460 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/383ce69044f049709627266e41ad430a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/383ce69044f049709627266e41ad430a 2023-04-28 14:59:15,464 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/383ce69044f049709627266e41ad430a, entries=150, sequenceid=180, filesize=11.8 K 2023-04-28 14:59:15,464 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 7fd2108ac9c35550edc28d90fafe4714 in 1240ms, sequenceid=180, compaction requested=true 2023-04-28 14:59:15,464 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:15,464 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:15,465 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:15,465 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:15,465 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,465 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:15,465 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:15,466 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,466 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:15,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:15,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:15,466 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:15,466 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:15,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:15,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:15,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:15,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:15,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:15,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c4e72c5[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:15,467 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:15,467 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:15,468 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:15,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:15,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:15,468 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:15,468 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:15,468 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:15,468 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,468 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:15,469 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:15,469 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,469 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:15,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:15,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:15,469 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:15,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:15,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:15,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:15,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:15,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:15,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:15,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:15,471 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:15,471 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:15,471 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:15,471 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:15,471 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:15,472 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:15,472 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:15,472 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:15,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:15,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:15,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:15,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:15,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:15,472 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:15,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:15,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:15,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:15,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:15,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:15,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:15,473 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:15,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:15,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,474 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,474 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:15,474 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:15,474 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:15,474 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:15,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:15,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:15,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:15,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:15,474 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:15,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:15,474 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:15,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:15,475 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:15,475 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:15,475 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:15,475 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:15,475 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:15,575 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:15,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:15,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:15,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:15,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:15,578 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:15,578 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:15,578 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:15,578 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:15,581 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:15,581 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:15,582 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,582 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:15,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:15,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:15,582 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,582 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:15,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:15,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:15,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:15,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:15,583 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:15,583 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:15,583 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:15,583 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:15,583 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:15,583 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:15,583 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:15,583 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:15,583 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:15,584 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:15,584 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7fd2108ac9c35550edc28d90fafe4714 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-28 14:59:15,584 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=A 2023-04-28 14:59:15,584 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:15,584 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=B 2023-04-28 14:59:15,584 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:15,584 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=C 2023-04-28 14:59:15,584 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:15,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:15,717 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:15,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:15,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694015729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:15,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52124 deadline: 1682694015729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:15,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694015735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:15,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694015735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,741 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:15,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694015736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,841 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:15,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:15,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694015837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52124 deadline: 1682694015837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:15,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694015837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:15,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694015837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:15,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52068 deadline: 1682694015842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:15,990 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/125223e820e8483db649fa29738d0629 2023-04-28 14:59:16,044 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:16,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52124 deadline: 1682694016042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,044 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:16,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52082 deadline: 1682694016043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,044 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:16,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694016043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,045 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:16,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694016043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:16,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52068 deadline: 1682694016044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:16,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52118 deadline: 1682694016345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:16,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52124 deadline: 1682694016346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:16,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52082 deadline: 1682694016346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:16,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694016346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:16,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52068 deadline: 1682694016349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,398 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/129f2a3d2cbe4c269f5c7782d789b646 2023-04-28 14:59:16,810 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/b184375536604da3b77fe64d55d91b52 2023-04-28 14:59:16,813 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/125223e820e8483db649fa29738d0629 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/125223e820e8483db649fa29738d0629 2023-04-28 14:59:16,816 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/125223e820e8483db649fa29738d0629, entries=150, sequenceid=200, filesize=11.8 K 2023-04-28 14:59:16,816 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/129f2a3d2cbe4c269f5c7782d789b646 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/129f2a3d2cbe4c269f5c7782d789b646 2023-04-28 14:59:16,819 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/129f2a3d2cbe4c269f5c7782d789b646, entries=150, sequenceid=200, filesize=11.8 K 2023-04-28 14:59:16,819 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/b184375536604da3b77fe64d55d91b52 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/b184375536604da3b77fe64d55d91b52 2023-04-28 14:59:16,821 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/b184375536604da3b77fe64d55d91b52, entries=150, sequenceid=200, filesize=11.8 K 2023-04-28 14:59:16,822 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 7fd2108ac9c35550edc28d90fafe4714 in 1238ms, sequenceid=200, compaction requested=true 2023-04-28 14:59:16,822 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:16,822 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:16,822 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:16,822 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:16,822 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,822 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:16,822 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:16,823 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:16,823 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:16,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:16,823 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:16,823 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:16,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:16,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:16,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:16,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:16,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:16,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c7792f7[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:16,824 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:16,824 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:16,825 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:16,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:16,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:16,825 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:16,825 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:16,825 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:16,825 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,825 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:16,826 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,826 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:16,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:16,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:16,826 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:16,826 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:16,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:16,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:16,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:16,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:16,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:16,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:16,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:16,827 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:16,827 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:16,827 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:16,827 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:16,827 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:16,828 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:16,828 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:16,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:16,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:16,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:16,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:16,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:16,828 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:16,829 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:16,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:16,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:16,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:16,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:16,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:16,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:16,829 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:16,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:16,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,831 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,831 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:16,831 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:16,831 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,831 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:16,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:16,831 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:16,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:16,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:16,831 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:16,831 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:16,831 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:16,831 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:16,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:16,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:16,831 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:16,831 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:16,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:16,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:16,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:16,856 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7fd2108ac9c35550edc28d90fafe4714 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-28 14:59:16,857 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=A 2023-04-28 14:59:16,857 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:16,857 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=B 2023-04-28 14:59:16,857 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:16,857 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=C 2023-04-28 14:59:16,857 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:16,883 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:16,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52124 deadline: 1682694016881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,889 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:16,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52082 deadline: 1682694016882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,889 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:16,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52118 deadline: 1682694016882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:16,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52068 deadline: 1682694016883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:16,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:52104 deadline: 1682694016883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,931 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:16,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:16,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:16,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:16,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:16,935 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:16,935 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:16,935 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:16,935 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:16,935 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:16,935 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:16,936 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:16,936 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:16,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:16,936 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:16,936 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:16,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:16,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:16,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:16,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:16,937 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:16,937 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:16,937 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:16,937 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:16,937 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:16,937 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:16,937 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:16,937 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:16,937 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:16,937 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:16,937 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:16,937 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:16,937 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:16,937 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,937 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:16,937 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:16,938 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,938 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:16,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:16,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:16,938 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:16,938 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:16,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:16,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:16,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:16,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:16,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:16,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@718721f6[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:16,939 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:16,939 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:16,940 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:16,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:16,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:16,940 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:16,940 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:16,940 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:16,940 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,940 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:16,941 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,941 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:16,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:16,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:16,941 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:16,941 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:16,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:16,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:16,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:16,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:16,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:16,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:16,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:16,942 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:16,942 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:16,943 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:16,943 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:16,943 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:16,943 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:16,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:16,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:16,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:16,943 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:16,944 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:16,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:16,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:16,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:16,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:16,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:16,944 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:16,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:16,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:16,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:16,945 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:16,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:16,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,946 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,946 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:16,946 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:16,946 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:16,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:16,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:16,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:16,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:16,946 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:16,946 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:16,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:16,946 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:16,946 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:16,947 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:16,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:16,947 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:16,947 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:16,947 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:16,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:16,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52124 deadline: 1682694016984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,993 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:16,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52082 deadline: 1682694016990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:16,994 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:16,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52118 deadline: 1682694016990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,000 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:17,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52068 deadline: 1682694016995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,001 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:17,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52104 deadline: 1682694016996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,047 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:17,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:17,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:17,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:17,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:17,049 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:17,049 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:17,049 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:17,049 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:17,050 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,050 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,050 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,050 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:17,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:17,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:17,051 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,051 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:17,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:17,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,051 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:17,051 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:17,051 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:17,051 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:17,051 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:17,051 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:17,052 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:17,052 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:17,052 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:17,052 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:17,052 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:17,052 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:17,052 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:17,052 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:17,052 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,052 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:17,052 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:17,053 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,053 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:17,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:17,054 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,054 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:17,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:17,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:17,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:17,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:17,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19af94d6[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:17,055 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:17,055 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,055 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,055 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:17,056 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:17,056 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,056 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:17,056 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:17,056 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,056 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:17,056 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:17,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:17,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:17,056 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:17,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:17,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:17,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:17,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:17,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:17,058 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:17,058 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:17,058 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:17,058 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:17,058 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:17,059 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,059 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:17,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:17,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,059 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:17,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:17,059 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:17,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:17,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:17,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,060 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:17,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,061 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:17,061 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:17,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:17,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:17,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:17,061 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:17,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:17,062 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:17,061 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:17,061 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,062 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:17,062 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,062 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,062 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,062 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,062 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:17,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:17,162 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:17,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:17,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:17,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:17,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:17,165 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:17,165 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:17,165 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:17,165 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:17,165 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,165 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,166 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:17,166 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:17,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:17,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,166 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,166 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:17,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:17,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,167 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:17,167 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:17,167 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:17,167 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:17,167 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:17,167 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:17,167 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:17,167 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:17,168 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:17,168 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:17,168 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:17,168 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:17,168 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:17,168 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:17,168 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,168 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:17,168 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:17,170 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,170 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:17,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:17,170 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,170 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:17,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:17,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:17,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:17,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:17,171 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:17,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36d8157c[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:17,171 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,172 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,172 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:17,172 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:17,172 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:17,172 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,172 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:17,172 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,172 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:17,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:17,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:17,172 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:17,173 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:17,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:17,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:17,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:17,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:17,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:17,174 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:17,174 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:17,174 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:17,174 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:17,175 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:17,175 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,175 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:17,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:17,175 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:17,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:17,176 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:17,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:17,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:17,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,178 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:17,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,179 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:17,179 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:17,179 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:17,180 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:17,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:17,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:17,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:17,180 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,180 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,180 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,180 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,180 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,180 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,180 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:17,180 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:17,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:17,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:17,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:17,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:17,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52124 deadline: 1682694017188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,200 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:17,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52118 deadline: 1682694017194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,200 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:17,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52082 deadline: 1682694017195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:17,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52068 deadline: 1682694017202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:17,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52104 deadline: 1682694017202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,261 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/e43699609bc642078717538125e4bfca 2023-04-28 14:59:17,269 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/cc1de880cd7b48f78753cdc817fbcff5 2023-04-28 14:59:17,280 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:17,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:17,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:17,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:17,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:17,283 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:17,283 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:17,283 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:17,283 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:17,284 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,284 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,284 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,284 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:17,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:17,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:17,285 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,285 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:17,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:17,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,285 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:17,285 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:17,285 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:17,285 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:17,285 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:17,286 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:17,286 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:17,286 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:17,286 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:17,286 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:17,286 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:17,286 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:17,286 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:17,286 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:17,286 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,286 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:17,286 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:17,287 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,287 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:17,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:17,287 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,287 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:17,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:17,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:17,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:17,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:17,288 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:17,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2628c97a[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:17,288 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,289 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,289 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:17,289 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,289 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:17,289 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:17,289 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:17,290 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:17,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:17,290 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:17,290 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:17,290 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:17,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:17,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:17,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:17,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:17,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:17,292 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:17,292 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:17,292 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:17,292 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:17,292 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:17,293 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,293 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:17,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:17,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,293 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:17,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:17,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:17,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:17,293 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:17,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,294 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:17,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,296 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,296 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:17,296 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,296 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:17,296 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:17,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:17,296 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,296 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,296 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,296 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,296 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:17,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:17,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:17,297 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:17,297 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:17,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:17,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:17,398 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:17,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:17,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:17,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:17,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:17,400 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:17,400 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:17,400 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:17,401 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:17,401 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,401 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,402 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,402 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:17,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:17,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:17,402 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,402 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:17,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:17,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,403 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:17,403 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:17,403 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:17,403 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:17,403 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:17,403 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:17,403 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:17,403 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:17,403 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:17,403 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:17,403 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:17,403 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:17,403 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:17,403 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:17,403 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,403 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:17,403 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:17,404 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,404 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:17,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:17,404 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,404 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:17,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:17,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:17,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:17,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:17,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ee8c4d8[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:17,405 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:17,405 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,405 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,405 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:17,405 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,406 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:17,405 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:17,406 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:17,407 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,407 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:17,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:17,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:17,407 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:17,407 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:17,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:17,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:17,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:17,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:17,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:17,408 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:17,408 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:17,408 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:17,408 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:17,408 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:17,409 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:17,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:17,409 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,409 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:17,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:17,409 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:17,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:17,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:17,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:17,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,411 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,412 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,412 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:17,412 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,412 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:17,412 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:17,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:17,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:17,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:17,412 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:17,412 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,412 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,412 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:17,412 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:17,412 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,412 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:17,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:17,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:17,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52124 deadline: 1682694017494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:17,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52118 deadline: 1682694017502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,506 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:17,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52082 deadline: 1682694017502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:17,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52068 deadline: 1682694017510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:17,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52104 deadline: 1682694017510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,512 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:17,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:17,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:17,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:17,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:17,515 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:17,515 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:17,515 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:17,515 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:17,515 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,515 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,516 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,516 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:17,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:17,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:17,516 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,516 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:17,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:17,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:17,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:17,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:17,517 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:17,517 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:17,517 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:17,517 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:17,517 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:17,517 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:17,517 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:17,517 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:17,517 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:17,517 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:17,517 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:17,517 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,517 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:17,517 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:17,518 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,518 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:17,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:17,518 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,518 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:17,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:17,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:17,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:17,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:17,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b84f4f7[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:17,519 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:17,519 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,520 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,520 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:17,520 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:17,520 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:17,520 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,520 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:17,521 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,521 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:17,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:17,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:17,521 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:17,521 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:17,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:17,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:17,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:17,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:17,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:17,522 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:17,522 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:17,522 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:17,522 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:17,522 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:17,523 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,523 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:17,523 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:17,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:17,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:17,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:17,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:17,524 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:17,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:17,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,525 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,526 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,526 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,526 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,526 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,526 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,526 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:17,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:17,526 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:17,526 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,526 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:17,526 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:17,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:17,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:17,527 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:17,527 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:17,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:17,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:17,627 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:17,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:17,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:17,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:17,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:17,629 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:17,629 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:17,629 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:17,629 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:17,630 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,630 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,630 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,630 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:17,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:17,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:17,631 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,631 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:17,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:17,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,631 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:17,631 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:17,632 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:17,632 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:17,632 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:17,632 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:17,632 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:17,632 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:17,632 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:17,632 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:17,632 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:17,632 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:17,632 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:17,632 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:17,632 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,632 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:17,632 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:17,633 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,633 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:17,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:17,633 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,633 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:17,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:17,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:17,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:17,634 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:17,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d81630e[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:17,634 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,635 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,635 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,635 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:17,635 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:17,635 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:17,635 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:17,636 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,636 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:17,636 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:17,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:17,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:17,636 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:17,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:17,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:17,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:17,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:17,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:17,637 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:17,637 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:17,637 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:17,637 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:17,637 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:17,638 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,638 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:17,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:17,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,638 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:17,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:17,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:17,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:17,639 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,640 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,641 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:17,641 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,641 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:17,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:17,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:17,641 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:17,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:17,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:17,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:17,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:17,641 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,642 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:17,642 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:17,642 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:17,642 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,642 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,642 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:17,642 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,676 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/791c3e0cb2564dc39db78ae1fa2315c3 2023-04-28 14:59:17,679 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/e43699609bc642078717538125e4bfca as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/e43699609bc642078717538125e4bfca 2023-04-28 14:59:17,682 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/e43699609bc642078717538125e4bfca, entries=200, sequenceid=213, filesize=14.1 K 2023-04-28 14:59:17,683 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/cc1de880cd7b48f78753cdc817fbcff5 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/cc1de880cd7b48f78753cdc817fbcff5 2023-04-28 14:59:17,685 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/cc1de880cd7b48f78753cdc817fbcff5, entries=150, sequenceid=213, filesize=11.8 K 2023-04-28 14:59:17,686 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/791c3e0cb2564dc39db78ae1fa2315c3 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/791c3e0cb2564dc39db78ae1fa2315c3 2023-04-28 14:59:17,688 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/791c3e0cb2564dc39db78ae1fa2315c3, entries=150, sequenceid=213, filesize=11.8 K 2023-04-28 14:59:17,689 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 7fd2108ac9c35550edc28d90fafe4714 in 833ms, sequenceid=213, compaction requested=true 2023-04-28 14:59:17,689 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:17,689 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:17,689 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-28 14:59:17,689 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-28 14:59:17,689 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:17,689 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:17,690 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 74864 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-28 14:59:17,691 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72474 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-28 14:59:17,691 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): 7fd2108ac9c35550edc28d90fafe4714/A is initiating minor compaction (all files) 2023-04-28 14:59:17,691 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 7fd2108ac9c35550edc28d90fafe4714/B is initiating minor compaction (all files) 2023-04-28 14:59:17,691 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7fd2108ac9c35550edc28d90fafe4714/A in TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:17,691 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7fd2108ac9c35550edc28d90fafe4714/B in TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:17,691 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/29779d727fa94e2fa235dc344e25fdb2, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/e02a9df85ab84487a8e12b6d7418bc94, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/e40bf8498de141f5bf6bd51aa1b866c7, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/a0aa5ab2e85646a1ac7cd2a0828c3682, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/129f2a3d2cbe4c269f5c7782d789b646, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/cc1de880cd7b48f78753cdc817fbcff5] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp, totalSize=70.8 K 2023-04-28 14:59:17,691 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/cb686a39049547249bfcc2530a797c6b, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/c09d8f4661ba460298d7cb221cd4eb5c, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/d6bb6029c1864285bb8fec687db20309, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/28e6c89bfbec476eb9ae3e404394c0c8, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/125223e820e8483db649fa29738d0629, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/e43699609bc642078717538125e4bfca] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp, totalSize=73.1 K 2023-04-28 14:59:17,691 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 29779d727fa94e2fa235dc344e25fdb2, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1682693949024 2023-04-28 14:59:17,691 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting cb686a39049547249bfcc2530a797c6b, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1682693949024 2023-04-28 14:59:17,691 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting e02a9df85ab84487a8e12b6d7418bc94, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1682693950165 2023-04-28 14:59:17,691 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting c09d8f4661ba460298d7cb221cd4eb5c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1682693950165 2023-04-28 14:59:17,692 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting e40bf8498de141f5bf6bd51aa1b866c7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1682693952323 2023-04-28 14:59:17,692 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting d6bb6029c1864285bb8fec687db20309, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1682693952323 2023-04-28 14:59:17,692 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting a0aa5ab2e85646a1ac7cd2a0828c3682, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1682693952944 2023-04-28 14:59:17,692 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 28e6c89bfbec476eb9ae3e404394c0c8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1682693952944 2023-04-28 14:59:17,692 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 129f2a3d2cbe4c269f5c7782d789b646, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1682693955096 2023-04-28 14:59:17,692 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 125223e820e8483db649fa29738d0629, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1682693955096 2023-04-28 14:59:17,692 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting cc1de880cd7b48f78753cdc817fbcff5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682693955729 2023-04-28 14:59:17,693 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting e43699609bc642078717538125e4bfca, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682693955729 2023-04-28 14:59:17,701 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 7fd2108ac9c35550edc28d90fafe4714#B#compaction#399 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:59:17,703 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7fd2108ac9c35550edc28d90fafe4714#A#compaction#400 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:59:17,711 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/79ce9681a6544ef7acdf16e88b47e686 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/79ce9681a6544ef7acdf16e88b47e686 2023-04-28 14:59:17,715 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 7fd2108ac9c35550edc28d90fafe4714/A of 7fd2108ac9c35550edc28d90fafe4714 into 79ce9681a6544ef7acdf16e88b47e686(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:17,715 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:17,715 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714., storeName=7fd2108ac9c35550edc28d90fafe4714/A, priority=10, startTime=1682693957689; duration=0sec 2023-04-28 14:59:17,715 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:17,715 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-28 14:59:17,716 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72474 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-28 14:59:17,716 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): 7fd2108ac9c35550edc28d90fafe4714/C is initiating minor compaction (all files) 2023-04-28 14:59:17,716 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7fd2108ac9c35550edc28d90fafe4714/C in TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:17,717 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/e9ffd2c947bf43cbae8231b08c845aaf, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/7b3dc46c66394f26b1ceb860b6d14cbe, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/da83122cb707457aa128a7a0fa89de42, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/383ce69044f049709627266e41ad430a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/b184375536604da3b77fe64d55d91b52, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/791c3e0cb2564dc39db78ae1fa2315c3] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp, totalSize=70.8 K 2023-04-28 14:59:17,717 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting e9ffd2c947bf43cbae8231b08c845aaf, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1682693949024 2023-04-28 14:59:17,717 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 7b3dc46c66394f26b1ceb860b6d14cbe, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1682693950165 2023-04-28 14:59:17,718 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting da83122cb707457aa128a7a0fa89de42, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1682693952323 2023-04-28 14:59:17,718 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 383ce69044f049709627266e41ad430a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1682693952944 2023-04-28 14:59:17,719 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting b184375536604da3b77fe64d55d91b52, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1682693955096 2023-04-28 14:59:17,719 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 791c3e0cb2564dc39db78ae1fa2315c3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682693955729 2023-04-28 14:59:17,727 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7fd2108ac9c35550edc28d90fafe4714#C#compaction#401 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:59:17,742 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:17,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:17,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:17,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:17,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:17,744 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:17,744 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:17,744 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:17,744 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:17,745 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,745 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,745 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:17,745 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:17,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:17,746 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:17,746 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:17,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:17,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:17,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:17,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:17,746 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:17,746 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:17,747 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:17,747 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:17,747 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:17,747 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:17,747 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:17,747 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:17,747 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:17,747 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7fd2108ac9c35550edc28d90fafe4714 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-28 14:59:17,747 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=A 2023-04-28 14:59:17,747 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:17,747 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=B 2023-04-28 14:59:17,747 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:17,747 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=C 2023-04-28 14:59:17,747 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:17,760 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/22232396f70e4b50ac4639d0ff5ee18e 2023-04-28 14:59:17,768 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/4814fda097584cd88c13f66e23acc0af 2023-04-28 14:59:18,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:18,010 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:18,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:18,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52104 deadline: 1682694018022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:18,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52124 deadline: 1682694018025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:18,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52118 deadline: 1682694018025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:18,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52082 deadline: 1682694018026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:18,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52068 deadline: 1682694018027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,110 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/d5e2abb3b780449f9d482e81b4ed1084 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/d5e2abb3b780449f9d482e81b4ed1084 2023-04-28 14:59:18,113 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 7fd2108ac9c35550edc28d90fafe4714/B of 7fd2108ac9c35550edc28d90fafe4714 into d5e2abb3b780449f9d482e81b4ed1084(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:18,113 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:18,114 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714., storeName=7fd2108ac9c35550edc28d90fafe4714/B, priority=10, startTime=1682693957689; duration=0sec 2023-04-28 14:59:18,114 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:18,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:18,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52104 deadline: 1682694018132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,134 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/f2bbd7903b9f41588811b5ab209d63d9 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/f2bbd7903b9f41588811b5ab209d63d9 2023-04-28 14:59:18,137 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 7fd2108ac9c35550edc28d90fafe4714/C of 7fd2108ac9c35550edc28d90fafe4714 into f2bbd7903b9f41588811b5ab209d63d9(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:18,137 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:18,137 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714., storeName=7fd2108ac9c35550edc28d90fafe4714/C, priority=10, startTime=1682693957689; duration=0sec 2023-04-28 14:59:18,137 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:18,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:18,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52124 deadline: 1682694018133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:18,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52118 deadline: 1682694018133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,142 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:18,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52082 deadline: 1682694018140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,142 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:18,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52068 deadline: 1682694018140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,175 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/9b7062457f1047a99ade9e8eba279f87 2023-04-28 14:59:18,179 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/22232396f70e4b50ac4639d0ff5ee18e as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/22232396f70e4b50ac4639d0ff5ee18e 2023-04-28 14:59:18,181 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/22232396f70e4b50ac4639d0ff5ee18e, entries=150, sequenceid=233, filesize=11.8 K 2023-04-28 14:59:18,182 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/4814fda097584cd88c13f66e23acc0af as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/4814fda097584cd88c13f66e23acc0af 2023-04-28 14:59:18,186 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/4814fda097584cd88c13f66e23acc0af, entries=150, sequenceid=233, filesize=11.8 K 2023-04-28 14:59:18,186 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/9b7062457f1047a99ade9e8eba279f87 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/9b7062457f1047a99ade9e8eba279f87 2023-04-28 14:59:18,189 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/9b7062457f1047a99ade9e8eba279f87, entries=150, sequenceid=233, filesize=11.8 K 2023-04-28 14:59:18,190 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 7fd2108ac9c35550edc28d90fafe4714 in 443ms, sequenceid=233, compaction requested=false 2023-04-28 14:59:18,190 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:18,190 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:18,190 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:18,190 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:18,190 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,190 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:18,190 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:18,191 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,191 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:18,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:18,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:18,191 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:18,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:18,191 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:18,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:18,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:18,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:18,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:18,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d0206dd[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:18,192 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:18,192 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:18,193 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:18,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:18,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:18,193 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:18,193 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:18,193 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,193 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:18,193 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:18,194 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:18,194 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,194 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:18,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:18,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:18,194 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:18,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:18,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:18,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:18,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:18,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:18,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:18,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:18,196 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:18,196 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:18,196 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:18,196 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:18,196 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:18,197 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:18,197 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:18,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:18,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:18,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:18,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:18,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:18,197 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:18,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:18,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:18,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:18,198 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:18,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:18,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:18,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:18,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,199 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:18,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:18,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,200 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:18,201 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:18,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:18,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:18,201 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,201 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:18,201 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:18,201 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:18,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:18,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:18,201 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,201 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:18,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:18,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:18,201 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:18,201 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:18,201 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:18,201 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:18,302 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:18,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:18,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:18,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:18,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:18,304 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:18,304 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:18,304 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:18,304 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:18,304 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:18,304 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:18,305 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:18,305 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:18,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:18,305 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,305 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:18,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:18,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:18,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:18,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:18,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:18,306 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:18,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:18,306 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:18,306 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:18,306 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:18,306 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:18,306 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:18,306 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:18,306 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:18,307 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7fd2108ac9c35550edc28d90fafe4714 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-28 14:59:18,307 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=A 2023-04-28 14:59:18,307 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:18,307 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=B 2023-04-28 14:59:18,307 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:18,307 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=C 2023-04-28 14:59:18,307 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:18,313 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/04699f51638c4662905ff3f1b993c3b5 2023-04-28 14:59:18,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:18,337 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:18,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:18,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52118 deadline: 1682694018357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,364 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:18,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52082 deadline: 1682694018358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,368 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:18,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52068 deadline: 1682694018360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,368 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:18,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52104 deadline: 1682694018361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,370 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:18,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52124 deadline: 1682694018363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:18,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52118 deadline: 1682694018464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:18,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52082 deadline: 1682694018464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:18,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52068 deadline: 1682694018469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:18,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52104 deadline: 1682694018469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:18,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52124 deadline: 1682694018471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:18,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52082 deadline: 1682694018668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:18,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52118 deadline: 1682694018668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:18,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52068 deadline: 1682694018672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:18,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52104 deadline: 1682694018672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:18,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52124 deadline: 1682694018677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,721 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/f9f2b396c0d6430daf0f847e8c1aeabe 2023-04-28 14:59:18,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:18,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52082 deadline: 1682694018973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,977 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:18,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52118 deadline: 1682694018975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:18,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52068 deadline: 1682694018977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:18,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52104 deadline: 1682694018978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:18,983 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:18,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52124 deadline: 1682694018981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,131 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/33bc18aeeae34e67a0bf03958fe064d8 2023-04-28 14:59:19,135 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/04699f51638c4662905ff3f1b993c3b5 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/04699f51638c4662905ff3f1b993c3b5 2023-04-28 14:59:19,137 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/04699f51638c4662905ff3f1b993c3b5, entries=150, sequenceid=248, filesize=11.8 K 2023-04-28 14:59:19,138 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/f9f2b396c0d6430daf0f847e8c1aeabe as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/f9f2b396c0d6430daf0f847e8c1aeabe 2023-04-28 14:59:19,140 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/f9f2b396c0d6430daf0f847e8c1aeabe, entries=150, sequenceid=248, filesize=11.8 K 2023-04-28 14:59:19,141 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/33bc18aeeae34e67a0bf03958fe064d8 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/33bc18aeeae34e67a0bf03958fe064d8 2023-04-28 14:59:19,144 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/33bc18aeeae34e67a0bf03958fe064d8, entries=150, sequenceid=248, filesize=11.8 K 2023-04-28 14:59:19,144 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 7fd2108ac9c35550edc28d90fafe4714 in 837ms, sequenceid=248, compaction requested=true 2023-04-28 14:59:19,144 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:19,144 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:19,144 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:19,144 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:19,144 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,145 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:19,145 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:19,145 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,145 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:19,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:19,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:19,146 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:19,146 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:19,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:19,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:19,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:19,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:19,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:19,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3564cfa1[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:19,147 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:19,147 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:19,147 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:19,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:19,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:19,147 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:19,147 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:19,147 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:19,147 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,148 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:19,148 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:19,148 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:19,148 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:19,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:19,148 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:19,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:19,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:19,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:19,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:19,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:19,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:19,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:19,150 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:19,150 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:19,150 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:19,150 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:19,150 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:19,151 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:19,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:19,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:19,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:19,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:19,151 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:19,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:19,152 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:19,152 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:19,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:19,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:19,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:19,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:19,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:19,152 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:19,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:19,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:19,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,154 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:19,154 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,154 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:19,154 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:19,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:19,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:19,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:19,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:19,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:19,154 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:19,154 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:19,154 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:19,154 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:19,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:19,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:19,154 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,154 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:19,154 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:19,154 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:19,254 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:19,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:19,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:19,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:19,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:19,257 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:19,257 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:19,257 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:19,257 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:19,257 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:19,257 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:19,258 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,258 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:19,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:19,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:19,258 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,258 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:19,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:19,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:19,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:19,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:19,259 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:19,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:19,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:19,259 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:19,259 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:19,259 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:19,259 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:19,259 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:19,259 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:19,259 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:19,259 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7fd2108ac9c35550edc28d90fafe4714 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-28 14:59:19,260 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=A 2023-04-28 14:59:19,260 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:19,260 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=B 2023-04-28 14:59:19,260 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:19,260 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=C 2023-04-28 14:59:19,260 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:19,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:19,479 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:19,492 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:19,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52082 deadline: 1682694019487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,492 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:19,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52118 deadline: 1682694019488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,494 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:19,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52124 deadline: 1682694019491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,497 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:19,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52104 deadline: 1682694019492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,497 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:19,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52068 deadline: 1682694019492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,597 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:19,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52082 deadline: 1682694019593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:19,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52118 deadline: 1682694019593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:19,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52124 deadline: 1682694019595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:19,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52104 deadline: 1682694019598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:19,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52068 deadline: 1682694019598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,666 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/48bc3fcf2ff143be9b7eb6ed99d5b72f 2023-04-28 14:59:19,802 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:19,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52082 deadline: 1682694019798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:19,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52118 deadline: 1682694019799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:19,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52124 deadline: 1682694019799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:19,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52104 deadline: 1682694019799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:19,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:19,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52068 deadline: 1682694019800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,074 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/2914d367da704a5f8a0cf07d30d45104 2023-04-28 14:59:20,080 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/696a910367014d49a2ea9567fe3a8373 2023-04-28 14:59:20,083 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/48bc3fcf2ff143be9b7eb6ed99d5b72f as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/48bc3fcf2ff143be9b7eb6ed99d5b72f 2023-04-28 14:59:20,085 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/48bc3fcf2ff143be9b7eb6ed99d5b72f, entries=150, sequenceid=267, filesize=11.9 K 2023-04-28 14:59:20,085 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/2914d367da704a5f8a0cf07d30d45104 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/2914d367da704a5f8a0cf07d30d45104 2023-04-28 14:59:20,088 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/2914d367da704a5f8a0cf07d30d45104, entries=150, sequenceid=267, filesize=11.9 K 2023-04-28 14:59:20,088 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/696a910367014d49a2ea9567fe3a8373 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/696a910367014d49a2ea9567fe3a8373 2023-04-28 14:59:20,091 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/696a910367014d49a2ea9567fe3a8373, entries=150, sequenceid=267, filesize=11.9 K 2023-04-28 14:59:20,091 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 7fd2108ac9c35550edc28d90fafe4714 in 832ms, sequenceid=267, compaction requested=true 2023-04-28 14:59:20,091 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:20,091 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:20,091 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:20,091 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:20,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:20,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:20,092 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:20,093 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,093 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:20,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:20,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19ebb6b6[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:20,094 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:20,094 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,094 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,094 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:20,094 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:20,094 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:20,095 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:20,095 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,095 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:20,095 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:20,095 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:20,095 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:20,095 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:20,095 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:20,095 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:20,095 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:20,095 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:20,096 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:20,096 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,096 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:20,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:20,096 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,097 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:20,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:20,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:20,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:20,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:20,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:20,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:20,099 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:20,099 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:20,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:20,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:20,099 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:20,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:20,108 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7fd2108ac9c35550edc28d90fafe4714 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-28 14:59:20,108 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=A 2023-04-28 14:59:20,108 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:20,108 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=B 2023-04-28 14:59:20,108 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:20,108 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=C 2023-04-28 14:59:20,108 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:20,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:20,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52082 deadline: 1682694020133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:20,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52118 deadline: 1682694020133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:20,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52124 deadline: 1682694020134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:20,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52104 deadline: 1682694020134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:20,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52068 deadline: 1682694020135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,199 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:20,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:20,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:20,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:20,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:20,202 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:20,202 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:20,202 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:20,202 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:20,202 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,202 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,203 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:20,203 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:20,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:20,203 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,203 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:20,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:20,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,204 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:20,204 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:20,204 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:20,204 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:20,204 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:20,204 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:20,205 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:20,205 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:20,205 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:20,205 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:20,205 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:20,205 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:20,205 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:20,205 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:20,205 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,205 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:20,205 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:20,206 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,206 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:20,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:20,206 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,206 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:20,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:20,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:20,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:20,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:20,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f843f96[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:20,207 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:20,207 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:20,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:20,208 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,208 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:20,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:20,209 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,209 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:20,209 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:20,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:20,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:20,209 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:20,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:20,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:20,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:20,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:20,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:20,211 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:20,211 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:20,211 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:20,211 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:20,211 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:20,212 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:20,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:20,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:20,212 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:20,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,213 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,217 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,217 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,217 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,217 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,217 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,217 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:20,217 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:20,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:20,217 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,217 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:20,217 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:20,222 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,222 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:20,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,222 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:20,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:20,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:20,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:20,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:20,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:20,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:20,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52082 deadline: 1682694020240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,247 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:20,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:20,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52118 deadline: 1682694020241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52124 deadline: 1682694020241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:20,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52104 deadline: 1682694020241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:20,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52068 deadline: 1682694020241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,317 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:20,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:20,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:20,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:20,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:20,320 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:20,320 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:20,320 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:20,320 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:20,320 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,320 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,321 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:20,321 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:20,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:20,322 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,322 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:20,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:20,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,323 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:20,324 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:20,324 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:20,324 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:20,324 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:20,324 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:20,324 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:20,324 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:20,324 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:20,324 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:20,324 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:20,324 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:20,324 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:20,324 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:20,324 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,324 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:20,324 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:20,325 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,325 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,326 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,326 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:20,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:20,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62bdf855[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:20,326 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:20,326 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,327 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,327 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:20,327 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:20,327 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,327 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:20,327 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:20,328 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:20,328 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:20,328 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,328 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:20,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:20,330 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:20,330 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:20,330 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:20,330 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:20,330 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:20,331 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,331 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:20,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:20,331 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:20,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:20,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:20,331 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:20,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:20,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,332 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:20,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,334 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:20,334 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:20,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:20,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:20,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:20,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:20,334 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,334 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:20,334 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,334 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:20,334 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:20,334 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:20,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:20,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:20,334 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,334 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,334 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,334 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,334 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,434 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:20,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:20,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:20,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:20,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:20,437 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:20,437 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:20,437 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:20,437 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:20,437 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,437 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,438 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,438 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:20,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:20,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:20,438 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,438 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:20,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:20,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,438 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:20,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:20,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:20,439 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:20,439 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:20,439 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:20,439 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:20,439 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:20,439 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:20,439 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:20,439 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:20,439 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:20,439 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:20,439 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:20,439 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,439 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:20,439 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:20,440 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,440 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:20,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:20,440 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,440 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:20,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:20,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:20,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:20,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:20,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71373a64[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:20,441 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:20,441 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,442 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,442 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:20,442 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:20,442 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:20,442 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,442 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:20,443 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:20,443 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:20,443 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:20,443 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:20,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:20,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:20,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:20,445 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:20,445 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:20,445 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:20,445 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:20,445 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:20,446 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:20,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:20,446 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,446 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:20,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:20,446 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:20,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:20,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:20,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,447 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:20,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,448 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,448 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,448 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,448 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,448 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:20,448 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:20,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:20,448 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:20,448 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,449 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,448 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:20,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:20,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:20,449 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:20,449 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:20,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:20,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:20,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:20,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52082 deadline: 1682694020447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:20,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52104 deadline: 1682694020450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:20,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52068 deadline: 1682694020450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:20,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52124 deadline: 1682694020450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:20,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52118 deadline: 1682694020450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,514 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/14c439238e5d4c9f8e55658cd1387ef4 2023-04-28 14:59:20,526 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/6651d3b7b66444b8b96ee5985ea826c5 2023-04-28 14:59:20,549 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:20,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:20,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:20,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:20,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:20,551 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:20,551 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:20,551 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:20,551 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:20,552 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,552 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,552 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,552 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:20,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:20,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:20,553 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,553 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:20,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:20,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:20,553 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:20,553 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:20,553 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:20,553 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:20,553 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:20,554 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:20,554 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:20,554 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:20,554 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:20,554 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:20,554 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:20,554 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:20,554 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:20,554 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,554 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:20,554 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:20,555 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,555 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:20,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:20,555 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,555 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:20,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:20,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:20,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:20,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:20,556 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:20,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@352d6cf2[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:20,556 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,557 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,557 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:20,557 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:20,557 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:20,558 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,558 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:20,558 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:20,558 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:20,558 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,558 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:20,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:20,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:20,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:20,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:20,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:20,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:20,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:20,560 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:20,560 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:20,560 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:20,560 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:20,560 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:20,561 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,561 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,561 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:20,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:20,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:20,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:20,562 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:20,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:20,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:20,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,562 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,565 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:20,565 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:20,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:20,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:20,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:20,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:20,565 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,565 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,565 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,565 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,565 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:20,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:20,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:20,565 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,565 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:20,565 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,565 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:20,565 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:20,666 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:20,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:20,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:20,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:20,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:20,668 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:20,668 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:20,668 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:20,668 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:20,677 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,678 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,678 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:20,678 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:20,679 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,679 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:20,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:20,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,679 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:20,679 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:20,679 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:20,680 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:20,680 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:20,680 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:20,680 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:20,680 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:20,680 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:20,680 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:20,680 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:20,680 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:20,680 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:20,680 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,680 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:20,680 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:20,681 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:20,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:20,681 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:20,681 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,681 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:20,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:20,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:20,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:20,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b9b9d4e[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:20,682 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:20,682 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,683 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,683 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:20,683 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:20,683 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:20,683 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,683 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:20,684 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,684 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:20,684 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:20,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:20,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:20,684 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:20,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:20,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:20,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:20,686 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:20,686 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:20,686 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:20,686 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:20,686 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:20,686 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:20,687 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,687 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:20,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:20,687 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:20,688 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:20,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,690 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,690 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,690 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,690 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,690 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,690 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,690 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:20,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:20,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:20,690 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,691 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:20,691 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:20,691 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:20,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:20,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:20,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:20,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:20,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:20,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:20,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:20,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52068 deadline: 1682694020753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:20,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52104 deadline: 1682694020754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:20,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52082 deadline: 1682694020755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,762 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:20,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52124 deadline: 1682694020757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,762 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:20,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52118 deadline: 1682694020758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,791 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:20,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:20,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:20,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:20,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:20,793 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:20,793 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:20,793 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:20,793 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:20,793 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,794 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,794 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:20,794 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:20,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:20,794 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,794 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:20,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:20,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:20,795 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:20,795 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:20,795 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:20,795 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:20,795 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:20,795 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:20,795 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:20,795 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:20,795 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:20,796 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:20,796 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:20,796 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:20,796 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:20,796 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,796 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:20,796 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:20,796 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,796 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:20,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:20,797 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,797 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:20,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:20,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:20,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:20,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:20,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76e9ad38[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:20,798 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:20,798 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,798 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,798 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:20,799 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:20,799 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:20,799 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,799 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:20,799 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,799 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:20,799 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:20,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:20,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:20,799 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:20,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:20,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:20,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:20,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:20,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:20,801 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:20,801 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:20,801 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:20,801 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:20,801 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:20,802 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,802 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:20,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:20,802 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:20,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:20,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:20,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:20,803 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:20,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,804 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:20,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,805 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,805 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,805 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:20,805 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,805 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:20,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:20,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:20,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:20,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:20,805 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:20,805 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:20,805 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,805 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:20,805 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,805 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:20,805 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:20,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:20,906 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:20,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:20,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:20,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:20,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:20,908 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:20,908 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:20,908 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:20,908 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:20,908 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,908 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,909 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,909 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:20,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:20,909 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,909 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,910 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:20,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:20,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:20,910 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:20,910 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:20,910 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:20,910 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:20,910 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:20,910 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:20,910 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:20,910 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:20,910 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:20,910 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:20,910 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:20,910 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,910 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:20,911 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:20,911 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,911 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:20,911 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,911 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:20,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:20,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ec54d9b[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:20,912 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:20,912 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,913 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,913 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:20,913 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:20,913 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:20,914 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,914 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:20,914 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:20,914 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:20,914 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:20,914 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:20,914 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:20,914 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:20,914 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:20,914 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:20,914 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:20,915 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,915 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:20,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:20,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,915 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:20,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:20,915 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:20,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:20,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:20,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,916 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:20,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:20,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,918 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,918 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:20,918 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:20,918 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:20,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:20,918 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:20,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:20,918 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:20,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:20,918 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:20,918 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:20,918 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,918 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:20,918 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:20,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:20,918 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:20,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:20,933 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/bfcc09a8f24547c68cd36a55a68c6f28 2023-04-28 14:59:20,936 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/14c439238e5d4c9f8e55658cd1387ef4 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/14c439238e5d4c9f8e55658cd1387ef4 2023-04-28 14:59:20,940 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/14c439238e5d4c9f8e55658cd1387ef4, entries=200, sequenceid=281, filesize=14.3 K 2023-04-28 14:59:20,941 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/6651d3b7b66444b8b96ee5985ea826c5 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/6651d3b7b66444b8b96ee5985ea826c5 2023-04-28 14:59:20,943 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/6651d3b7b66444b8b96ee5985ea826c5, entries=150, sequenceid=281, filesize=11.9 K 2023-04-28 14:59:20,944 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/bfcc09a8f24547c68cd36a55a68c6f28 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/bfcc09a8f24547c68cd36a55a68c6f28 2023-04-28 14:59:20,947 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/bfcc09a8f24547c68cd36a55a68c6f28, entries=150, sequenceid=281, filesize=11.9 K 2023-04-28 14:59:20,947 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 7fd2108ac9c35550edc28d90fafe4714 in 839ms, sequenceid=281, compaction requested=true 2023-04-28 14:59:20,947 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:20,947 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:20,947 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 14:59:20,947 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 14:59:20,947 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:20,948 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:20,948 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 63432 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 14:59:20,948 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60992 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 14:59:20,949 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): 7fd2108ac9c35550edc28d90fafe4714/A is initiating minor compaction (all files) 2023-04-28 14:59:20,949 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 7fd2108ac9c35550edc28d90fafe4714/B is initiating minor compaction (all files) 2023-04-28 14:59:20,949 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7fd2108ac9c35550edc28d90fafe4714/A in TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:20,949 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7fd2108ac9c35550edc28d90fafe4714/B in TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:20,949 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/79ce9681a6544ef7acdf16e88b47e686, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/22232396f70e4b50ac4639d0ff5ee18e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/04699f51638c4662905ff3f1b993c3b5, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/48bc3fcf2ff143be9b7eb6ed99d5b72f, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/14c439238e5d4c9f8e55658cd1387ef4] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp, totalSize=61.9 K 2023-04-28 14:59:20,949 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/d5e2abb3b780449f9d482e81b4ed1084, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/4814fda097584cd88c13f66e23acc0af, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/f9f2b396c0d6430daf0f847e8c1aeabe, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/2914d367da704a5f8a0cf07d30d45104, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/6651d3b7b66444b8b96ee5985ea826c5] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp, totalSize=59.6 K 2023-04-28 14:59:20,949 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 79ce9681a6544ef7acdf16e88b47e686, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682693955729 2023-04-28 14:59:20,949 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting d5e2abb3b780449f9d482e81b4ed1084, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682693955729 2023-04-28 14:59:20,949 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 22232396f70e4b50ac4639d0ff5ee18e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1682693956880 2023-04-28 14:59:20,949 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 4814fda097584cd88c13f66e23acc0af, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1682693956880 2023-04-28 14:59:20,949 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 04699f51638c4662905ff3f1b993c3b5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1682693958020 2023-04-28 14:59:20,949 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting f9f2b396c0d6430daf0f847e8c1aeabe, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1682693958020 2023-04-28 14:59:20,949 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 48bc3fcf2ff143be9b7eb6ed99d5b72f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682693958357 2023-04-28 14:59:20,950 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 2914d367da704a5f8a0cf07d30d45104, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682693958357 2023-04-28 14:59:20,950 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 14c439238e5d4c9f8e55658cd1387ef4, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1682693959490 2023-04-28 14:59:20,950 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 6651d3b7b66444b8b96ee5985ea826c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1682693959490 2023-04-28 14:59:20,958 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7fd2108ac9c35550edc28d90fafe4714#A#compaction#414 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:59:20,963 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 7fd2108ac9c35550edc28d90fafe4714#B#compaction#415 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:59:20,970 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/3360e845ce3b497280a0e21a3fb06752 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/3360e845ce3b497280a0e21a3fb06752 2023-04-28 14:59:20,970 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/292e94dec0104d82ae9cac7e57e8aaef as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/292e94dec0104d82ae9cac7e57e8aaef 2023-04-28 14:59:20,974 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 7fd2108ac9c35550edc28d90fafe4714/B of 7fd2108ac9c35550edc28d90fafe4714 into 292e94dec0104d82ae9cac7e57e8aaef(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:20,974 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 7fd2108ac9c35550edc28d90fafe4714/A of 7fd2108ac9c35550edc28d90fafe4714 into 3360e845ce3b497280a0e21a3fb06752(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:20,974 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:20,974 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:20,974 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714., storeName=7fd2108ac9c35550edc28d90fafe4714/B, priority=11, startTime=1682693960947; duration=0sec 2023-04-28 14:59:20,974 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714., storeName=7fd2108ac9c35550edc28d90fafe4714/A, priority=11, startTime=1682693960947; duration=0sec 2023-04-28 14:59:20,974 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:20,974 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:20,974 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 14:59:20,976 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60992 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 14:59:20,976 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 7fd2108ac9c35550edc28d90fafe4714/C is initiating minor compaction (all files) 2023-04-28 14:59:20,976 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7fd2108ac9c35550edc28d90fafe4714/C in TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:20,976 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/f2bbd7903b9f41588811b5ab209d63d9, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/9b7062457f1047a99ade9e8eba279f87, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/33bc18aeeae34e67a0bf03958fe064d8, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/696a910367014d49a2ea9567fe3a8373, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/bfcc09a8f24547c68cd36a55a68c6f28] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp, totalSize=59.6 K 2023-04-28 14:59:20,976 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting f2bbd7903b9f41588811b5ab209d63d9, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682693955729 2023-04-28 14:59:20,977 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 9b7062457f1047a99ade9e8eba279f87, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1682693956880 2023-04-28 14:59:20,977 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 33bc18aeeae34e67a0bf03958fe064d8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1682693958020 2023-04-28 14:59:20,977 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 696a910367014d49a2ea9567fe3a8373, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682693958357 2023-04-28 14:59:20,977 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting bfcc09a8f24547c68cd36a55a68c6f28, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1682693959490 2023-04-28 14:59:20,983 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 7fd2108ac9c35550edc28d90fafe4714#C#compaction#416 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:59:21,018 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:21,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:21,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:21,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:21,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:21,021 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:21,021 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:21,021 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:21,021 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:21,021 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:21,022 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:21,022 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:21,022 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:21,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:21,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:21,022 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:21,022 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:21,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:21,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:21,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:21,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:21,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:21,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:21,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:21,023 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:21,023 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:21,023 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:21,024 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:21,024 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:21,024 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:21,024 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:21,024 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7fd2108ac9c35550edc28d90fafe4714 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-28 14:59:21,024 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=A 2023-04-28 14:59:21,024 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:21,024 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=B 2023-04-28 14:59:21,024 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:21,024 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=C 2023-04-28 14:59:21,024 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:21,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:21,265 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:21,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:21,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52104 deadline: 1682694021271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:21,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:21,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52124 deadline: 1682694021272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:21,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:21,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52082 deadline: 1682694021272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:21,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:21,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52068 deadline: 1682694021273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:21,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:21,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52118 deadline: 1682694021274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:21,377 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:21,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52104 deadline: 1682694021375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:21,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:21,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52124 deadline: 1682694021379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:21,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:21,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52082 deadline: 1682694021379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:21,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:21,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:21,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52068 deadline: 1682694021379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:21,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52118 deadline: 1682694021379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:21,389 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/8b88a20deb9c45dd8bf8ecc083048036 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/8b88a20deb9c45dd8bf8ecc083048036 2023-04-28 14:59:21,392 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 7fd2108ac9c35550edc28d90fafe4714/C of 7fd2108ac9c35550edc28d90fafe4714 into 8b88a20deb9c45dd8bf8ecc083048036(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:21,392 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:21,392 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714., storeName=7fd2108ac9c35550edc28d90fafe4714/C, priority=11, startTime=1682693960948; duration=0sec 2023-04-28 14:59:21,392 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:21,438 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/ea93cf5267374e5bbf12d0a6321330ca 2023-04-28 14:59:21,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:21,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52104 deadline: 1682694021579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:21,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:21,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52068 deadline: 1682694021585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:21,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:21,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52124 deadline: 1682694021585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:21,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:21,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52082 deadline: 1682694021586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:21,589 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:21,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52118 deadline: 1682694021586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:21,851 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/cf4f505914ff43be854dabf3e38d5c03 2023-04-28 14:59:21,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:21,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52104 deadline: 1682694021882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:21,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:21,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52124 deadline: 1682694021887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:21,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:21,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52082 deadline: 1682694021887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:21,894 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:21,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52068 deadline: 1682694021889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:21,894 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:21,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52118 deadline: 1682694021890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,259 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/c97c1901e40946ae8d2c3bea31b3f79d 2023-04-28 14:59:22,263 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/ea93cf5267374e5bbf12d0a6321330ca as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/ea93cf5267374e5bbf12d0a6321330ca 2023-04-28 14:59:22,265 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/ea93cf5267374e5bbf12d0a6321330ca, entries=150, sequenceid=302, filesize=11.9 K 2023-04-28 14:59:22,266 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/cf4f505914ff43be854dabf3e38d5c03 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/cf4f505914ff43be854dabf3e38d5c03 2023-04-28 14:59:22,268 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/cf4f505914ff43be854dabf3e38d5c03, entries=150, sequenceid=302, filesize=11.9 K 2023-04-28 14:59:22,269 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/c97c1901e40946ae8d2c3bea31b3f79d as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/c97c1901e40946ae8d2c3bea31b3f79d 2023-04-28 14:59:22,271 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/c97c1901e40946ae8d2c3bea31b3f79d, entries=150, sequenceid=302, filesize=11.9 K 2023-04-28 14:59:22,272 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 7fd2108ac9c35550edc28d90fafe4714 in 1248ms, sequenceid=302, compaction requested=false 2023-04-28 14:59:22,272 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:22,272 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:22,272 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:22,272 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:22,272 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,272 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:22,272 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:22,274 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:22,274 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:22,274 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:22,274 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:22,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:22,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60d1fc4d[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:22,275 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:22,276 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:22,276 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:22,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:22,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:22,276 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:22,276 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,276 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:22,276 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:22,276 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:22,277 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:22,277 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,277 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:22,277 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:22,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:22,279 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:22,279 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:22,279 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:22,279 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:22,279 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:22,280 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:22,280 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:22,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:22,280 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:22,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:22,280 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:22,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:22,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:22,281 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:22,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:22,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:22,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,282 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:22,282 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:22,282 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:22,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:22,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:22,283 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:22,283 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,283 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:22,283 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:22,283 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:22,283 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:22,283 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:22,283 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:22,283 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:22,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:22,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:22,383 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:22,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:22,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:22,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:22,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:22,389 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:22,389 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:22,389 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:22,390 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:22,390 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:22,390 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:22,391 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:22,391 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:22,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:22,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:22,391 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,391 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:22,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:22,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:22,391 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7fd2108ac9c35550edc28d90fafe4714 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-28 14:59:22,392 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=A 2023-04-28 14:59:22,392 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:22,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:22,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:22,392 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=B 2023-04-28 14:59:22,392 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:22,392 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:22,392 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=C 2023-04-28 14:59:22,392 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:22,399 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:22,399 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:22,399 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:22,399 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:22,399 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:22,399 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:22,399 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:22,399 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:22,399 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:22,400 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:22,400 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:22,400 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:22,400 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:22,400 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,400 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:22,400 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:22,401 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:22,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:22,402 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:22,402 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:22,402 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:22,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:22,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47876cf4[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:22,404 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:22,404 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:22,414 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:22,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:22,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:22,415 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:22,415 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:22,415 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:22,415 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,415 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:22,416 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:22,416 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:22,416 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:22,416 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:22,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:22,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:22,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:22,418 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:22,418 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:22,418 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:22,418 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:22,418 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:22,420 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:22,420 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:22,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:22,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:22,420 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:22,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:22,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:22,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:22,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:22,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:22,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:22,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:22,421 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:22,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:22,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,422 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:22,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:22,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,424 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,424 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:22,424 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:22,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:22,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:22,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:22,424 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:22,424 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:22,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:22,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:22,424 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:22,424 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,424 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:22,424 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:22,424 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:22,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:22,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:22,424 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:22,424 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:22,431 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:22,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52118 deadline: 1682694022422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:22,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52104 deadline: 1682694022429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,437 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:22,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52082 deadline: 1682694022430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,437 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:22,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52124 deadline: 1682694022430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,441 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:22,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52068 deadline: 1682694022432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,525 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:22,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:22,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:22,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:22,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:22,528 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:22,528 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:22,528 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:22,528 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:22,528 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:22,528 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:22,529 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,529 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:22,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:22,529 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,529 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:22,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:22,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:22,530 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:22,530 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:22,530 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:22,530 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:22,530 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:22,530 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:22,530 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:22,530 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:22,530 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. started... 2023-04-28 14:59:22,530 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. as already flushing 2023-04-28 14:59:22,530 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:22,530 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:22,530 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:22,530 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,530 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:22,530 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:22,531 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:22,531 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:22,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:22,531 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:22,531 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:22,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:22,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:22,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:22,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:22,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:22,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cbc8951[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:22,532 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:22,532 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:22,532 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:22,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:22,533 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:22,533 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:22,533 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,533 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:22,533 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:22,533 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,533 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:22,533 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:22,533 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:22,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:22,535 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:22,536 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:22,536 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:22,536 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:22,536 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:22,536 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:22,536 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:22,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:22,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:22,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:22,536 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:22,537 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:22,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:22,538 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,539 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,539 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:22,539 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:22,539 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:22,539 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,539 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:22,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:22,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:22,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:22,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:22,539 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:22,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:22,539 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:22,539 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:22,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52118 deadline: 1682694022532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,539 DEBUG [Thread-2765] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:22,539 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:22,540 DEBUG [Thread-2765] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:22,540 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:22,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:22,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:22,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:22,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52104 deadline: 1682694022536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:22,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52082 deadline: 1682694022538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:22,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52124 deadline: 1682694022538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,548 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:22,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52068 deadline: 1682694022542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,628 DEBUG [Thread-2772] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3c9f61fa to 127.0.0.1:61762 2023-04-28 14:59:22,628 DEBUG [Thread-2772] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:59:22,628 DEBUG [Thread-2766] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x31fd1c1f to 127.0.0.1:61762 2023-04-28 14:59:22,628 DEBUG [Thread-2766] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:59:22,630 DEBUG [Thread-2774] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x534826c2 to 127.0.0.1:61762 2023-04-28 14:59:22,630 DEBUG [Thread-2774] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:59:22,632 DEBUG [Thread-2770] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5345748a to 127.0.0.1:61762 2023-04-28 14:59:22,632 DEBUG [Thread-2770] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:59:22,634 DEBUG [Thread-2768] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x209faa3c to 127.0.0.1:61762 2023-04-28 14:59:22,634 DEBUG [Thread-2768] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:59:22,640 DEBUG [Thread-2765] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:22,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:22,742 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:22,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52118 deadline: 1682694022741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,745 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:22,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52082 deadline: 1682694022745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,746 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:22,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52104 deadline: 1682694022745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:22,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52124 deadline: 1682694022746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:22,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52068 deadline: 1682694022750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:22,816 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/4f961dd0b9744f51a9df3f1c6301a084 2023-04-28 14:59:23,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:23,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52118 deadline: 1682694023044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:23,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:23,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52082 deadline: 1682694023047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:23,050 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:23,050 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:23,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52104 deadline: 1682694023050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:23,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52124 deadline: 1682694023050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:23,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:23,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52068 deadline: 1682694023054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:23,239 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/a34017a0f735434185fc8574bd2ade7e 2023-04-28 14:59:23,550 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:23,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:23,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52118 deadline: 1682694023549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:23,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52082 deadline: 1682694023550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:23,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:23,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52104 deadline: 1682694023552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:23,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:23,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52124 deadline: 1682694023553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:23,558 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:23,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:52068 deadline: 1682694023558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:23,655 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/01975acccd034a4ea0e34a02466c0a8a 2023-04-28 14:59:23,658 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/4f961dd0b9744f51a9df3f1c6301a084 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/4f961dd0b9744f51a9df3f1c6301a084 2023-04-28 14:59:23,661 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/4f961dd0b9744f51a9df3f1c6301a084, entries=200, sequenceid=317, filesize=14.3 K 2023-04-28 14:59:23,662 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/a34017a0f735434185fc8574bd2ade7e as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/a34017a0f735434185fc8574bd2ade7e 2023-04-28 14:59:23,666 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/a34017a0f735434185fc8574bd2ade7e, entries=150, sequenceid=317, filesize=11.9 K 2023-04-28 14:59:23,666 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/01975acccd034a4ea0e34a02466c0a8a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/01975acccd034a4ea0e34a02466c0a8a 2023-04-28 14:59:23,670 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/01975acccd034a4ea0e34a02466c0a8a, entries=150, sequenceid=317, filesize=11.9 K 2023-04-28 14:59:23,671 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 7fd2108ac9c35550edc28d90fafe4714 in 1280ms, sequenceid=317, compaction requested=true 2023-04-28 14:59:23,671 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:23,671 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:23,671 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:59:23,672 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:59:23,671 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:23,672 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:23,672 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39680 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:59:23,673 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): 7fd2108ac9c35550edc28d90fafe4714/A is initiating minor compaction (all files) 2023-04-28 14:59:23,673 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:59:23,673 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7fd2108ac9c35550edc28d90fafe4714/A in TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:23,673 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 7fd2108ac9c35550edc28d90fafe4714/B is initiating minor compaction (all files) 2023-04-28 14:59:23,673 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/3360e845ce3b497280a0e21a3fb06752, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/ea93cf5267374e5bbf12d0a6321330ca, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/4f961dd0b9744f51a9df3f1c6301a084] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp, totalSize=38.8 K 2023-04-28 14:59:23,673 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7fd2108ac9c35550edc28d90fafe4714/B in TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:23,673 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/292e94dec0104d82ae9cac7e57e8aaef, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/cf4f505914ff43be854dabf3e38d5c03, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/a34017a0f735434185fc8574bd2ade7e] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp, totalSize=36.4 K 2023-04-28 14:59:23,673 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 3360e845ce3b497280a0e21a3fb06752, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1682693959490 2023-04-28 14:59:23,673 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 292e94dec0104d82ae9cac7e57e8aaef, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1682693959490 2023-04-28 14:59:23,673 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting ea93cf5267374e5bbf12d0a6321330ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1682693960133 2023-04-28 14:59:23,674 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting cf4f505914ff43be854dabf3e38d5c03, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1682693960133 2023-04-28 14:59:23,674 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 4f961dd0b9744f51a9df3f1c6301a084, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1682693961267 2023-04-28 14:59:23,674 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting a34017a0f735434185fc8574bd2ade7e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1682693961267 2023-04-28 14:59:23,680 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7fd2108ac9c35550edc28d90fafe4714#A#compaction#423 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:59:23,680 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 7fd2108ac9c35550edc28d90fafe4714#B#compaction#424 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:59:24,097 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/7157c3b1547f4ac8a3b456bb480ee5ed as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/7157c3b1547f4ac8a3b456bb480ee5ed 2023-04-28 14:59:24,097 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/6513fae5236a46969710ebeeb58c67f3 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/6513fae5236a46969710ebeeb58c67f3 2023-04-28 14:59:24,102 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7fd2108ac9c35550edc28d90fafe4714/B of 7fd2108ac9c35550edc28d90fafe4714 into 7157c3b1547f4ac8a3b456bb480ee5ed(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:24,102 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:24,102 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714., storeName=7fd2108ac9c35550edc28d90fafe4714/B, priority=13, startTime=1682693963671; duration=0sec 2023-04-28 14:59:24,102 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:24,102 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:59:24,102 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7fd2108ac9c35550edc28d90fafe4714/A of 7fd2108ac9c35550edc28d90fafe4714 into 6513fae5236a46969710ebeeb58c67f3(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:24,102 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:24,102 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714., storeName=7fd2108ac9c35550edc28d90fafe4714/A, priority=13, startTime=1682693963671; duration=0sec 2023-04-28 14:59:24,102 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:24,103 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:59:24,103 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 7fd2108ac9c35550edc28d90fafe4714/C is initiating minor compaction (all files) 2023-04-28 14:59:24,103 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7fd2108ac9c35550edc28d90fafe4714/C in TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:24,103 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/8b88a20deb9c45dd8bf8ecc083048036, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/c97c1901e40946ae8d2c3bea31b3f79d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/01975acccd034a4ea0e34a02466c0a8a] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp, totalSize=36.4 K 2023-04-28 14:59:24,104 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 8b88a20deb9c45dd8bf8ecc083048036, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1682693959490 2023-04-28 14:59:24,104 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting c97c1901e40946ae8d2c3bea31b3f79d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1682693960133 2023-04-28 14:59:24,104 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 01975acccd034a4ea0e34a02466c0a8a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1682693961267 2023-04-28 14:59:24,111 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 7fd2108ac9c35550edc28d90fafe4714#C#compaction#425 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:59:24,525 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/768fd6a750ab4619a8411f89addd65c4 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/768fd6a750ab4619a8411f89addd65c4 2023-04-28 14:59:24,531 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7fd2108ac9c35550edc28d90fafe4714/C of 7fd2108ac9c35550edc28d90fafe4714 into 768fd6a750ab4619a8411f89addd65c4(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:24,531 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:24,531 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714., storeName=7fd2108ac9c35550edc28d90fafe4714/C, priority=13, startTime=1682693963672; duration=0sec 2023-04-28 14:59:24,531 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:24,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:24,556 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7fd2108ac9c35550edc28d90fafe4714 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-28 14:59:24,557 DEBUG [Thread-2759] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x58ae18e2 to 127.0.0.1:61762 2023-04-28 14:59:24,557 DEBUG [Thread-2759] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:59:24,557 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=A 2023-04-28 14:59:24,557 DEBUG [Thread-2757] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x647f95f6 to 127.0.0.1:61762 2023-04-28 14:59:24,557 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:24,557 DEBUG [Thread-2757] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:59:24,558 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=B 2023-04-28 14:59:24,558 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:24,558 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=C 2023-04-28 14:59:24,558 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:24,560 DEBUG [Thread-2755] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06846250 to 127.0.0.1:61762 2023-04-28 14:59:24,561 DEBUG [Thread-2755] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:59:24,564 DEBUG [Thread-2761] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x029c1e76 to 127.0.0.1:61762 2023-04-28 14:59:24,564 DEBUG [Thread-2761] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:59:24,564 DEBUG [Thread-2763] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c240146 to 127.0.0.1:61762 2023-04-28 14:59:24,564 DEBUG [Thread-2763] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:59:24,564 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-28 14:59:24,565 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(394): wrote 58 2023-04-28 14:59:24,565 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(394): wrote 54 2023-04-28 14:59:24,565 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(394): wrote 48 2023-04-28 14:59:24,565 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(394): wrote 57 2023-04-28 14:59:24,565 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(394): wrote 50 2023-04-28 14:59:24,565 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-28 14:59:24,565 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-28 14:59:24,565 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(402): scanned 2359 2023-04-28 14:59:24,565 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(403): verified 7070 rows 2023-04-28 14:59:24,565 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(402): scanned 2334 2023-04-28 14:59:24,566 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(403): verified 7002 rows 2023-04-28 14:59:24,566 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(402): scanned 2350 2023-04-28 14:59:24,566 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(403): verified 7050 rows 2023-04-28 14:59:24,566 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(402): scanned 2352 2023-04-28 14:59:24,566 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(403): verified 7056 rows 2023-04-28 14:59:24,566 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(402): scanned 2344 2023-04-28 14:59:24,566 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(403): verified 7029 rows 2023-04-28 14:59:24,566 INFO [Listener at localhost.localdomain/40193] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-28 14:59:24,566 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a885842 to 127.0.0.1:61762 2023-04-28 14:59:24,566 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:59:24,568 INFO [Listener at localhost.localdomain/40193] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-28 14:59:24,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.HMaster$11(2418): Client=jenkins//136.243.18.41 disable TestAcidGuarantees 2023-04-28 14:59:24,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-28 14:59:24,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-28 14:59:24,572 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682693964571"}]},"ts":"1682693964571"} 2023-04-28 14:59:24,572 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-28 14:59:24,573 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-28 14:59:24,573 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7fd2108ac9c35550edc28d90fafe4714, UNASSIGN}] 2023-04-28 14:59:24,574 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7fd2108ac9c35550edc28d90fafe4714, UNASSIGN 2023-04-28 14:59:24,574 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=7fd2108ac9c35550edc28d90fafe4714, regionState=CLOSING, regionLocation=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:24,574 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693964574"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682693964574"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682693964574"}]},"ts":"1682693964574"} 2023-04-28 14:59:24,575 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482}] 2023-04-28 14:59:24,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-28 14:59:24,727 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(109): Close 7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:24,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1604): Closing 7fd2108ac9c35550edc28d90fafe4714, disabling compactions & flushes 2023-04-28 14:59:24,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:24,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-28 14:59:24,970 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/12ccd5ab53434a2bb7b55cacf9f082b7 2023-04-28 14:59:25,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-28 14:59:25,390 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/445d09612222469383b9b2bd066c3a1f 2023-04-28 14:59:25,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-28 14:59:25,809 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/692d5bf315284269a579ad896ec56e29 2023-04-28 14:59:25,820 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/12ccd5ab53434a2bb7b55cacf9f082b7 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/12ccd5ab53434a2bb7b55cacf9f082b7 2023-04-28 14:59:25,827 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/12ccd5ab53434a2bb7b55cacf9f082b7, entries=150, sequenceid=340, filesize=11.9 K 2023-04-28 14:59:25,827 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/445d09612222469383b9b2bd066c3a1f as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/445d09612222469383b9b2bd066c3a1f 2023-04-28 14:59:25,833 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/445d09612222469383b9b2bd066c3a1f, entries=150, sequenceid=340, filesize=11.9 K 2023-04-28 14:59:25,834 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/692d5bf315284269a579ad896ec56e29 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/692d5bf315284269a579ad896ec56e29 2023-04-28 14:59:25,840 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/692d5bf315284269a579ad896ec56e29, entries=150, sequenceid=340, filesize=11.9 K 2023-04-28 14:59:25,841 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=20.13 KB/20610 for 7fd2108ac9c35550edc28d90fafe4714 in 1285ms, sequenceid=340, compaction requested=false 2023-04-28 14:59:25,841 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:25,841 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:25,841 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:25,841 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. after waiting 0 ms 2023-04-28 14:59:25,841 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:25,841 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2745): Flushing 7fd2108ac9c35550edc28d90fafe4714 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-28 14:59:25,841 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=A 2023-04-28 14:59:25,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:25,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=B 2023-04-28 14:59:25,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:25,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7fd2108ac9c35550edc28d90fafe4714, store=C 2023-04-28 14:59:25,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:26,252 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=346 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/78ae3fd7555148d2a4314dc191b373fd 2023-04-28 14:59:26,270 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=346 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/eeff97bfbd9247c3ae8085bfe4221a66 2023-04-28 14:59:26,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-28 14:59:26,682 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=346 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/7d8ca8b71caa45f9945b551a615b2787 2023-04-28 14:59:26,688 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/A/78ae3fd7555148d2a4314dc191b373fd as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/78ae3fd7555148d2a4314dc191b373fd 2023-04-28 14:59:26,693 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/78ae3fd7555148d2a4314dc191b373fd, entries=50, sequenceid=346, filesize=7.1 K 2023-04-28 14:59:26,694 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/B/eeff97bfbd9247c3ae8085bfe4221a66 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/eeff97bfbd9247c3ae8085bfe4221a66 2023-04-28 14:59:26,699 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/eeff97bfbd9247c3ae8085bfe4221a66, entries=50, sequenceid=346, filesize=7.1 K 2023-04-28 14:59:26,700 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/.tmp/C/7d8ca8b71caa45f9945b551a615b2787 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/7d8ca8b71caa45f9945b551a615b2787 2023-04-28 14:59:26,705 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/7d8ca8b71caa45f9945b551a615b2787, entries=50, sequenceid=346, filesize=7.1 K 2023-04-28 14:59:26,706 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 7fd2108ac9c35550edc28d90fafe4714 in 865ms, sequenceid=346, compaction requested=true 2023-04-28 14:59:26,724 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/8c095c95c9ef48a3acdd6c8b91e07e4f, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/ee2438536df941fb984ac1ff91cafc01, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/3109db1b49e3434497771ef287f1faa9, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/b8840ab117f44adaae6ba92611caa7cd, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/b03b201dc1b34eb09a0de3efd6405581, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/37e761bfd59d480eaf40a4350f34914d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/5a9596747a664d78852391077d56fe98, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/81a86d93e59c4910b77b25f268bbc5b2, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/4e8d1b80884b464f81edb7b632c0d0ad, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/cb686a39049547249bfcc2530a797c6b, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/c09d8f4661ba460298d7cb221cd4eb5c, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/d6bb6029c1864285bb8fec687db20309, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/28e6c89bfbec476eb9ae3e404394c0c8, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/125223e820e8483db649fa29738d0629, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/e43699609bc642078717538125e4bfca, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/79ce9681a6544ef7acdf16e88b47e686, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/22232396f70e4b50ac4639d0ff5ee18e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/04699f51638c4662905ff3f1b993c3b5, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/48bc3fcf2ff143be9b7eb6ed99d5b72f, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/14c439238e5d4c9f8e55658cd1387ef4, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/3360e845ce3b497280a0e21a3fb06752, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/ea93cf5267374e5bbf12d0a6321330ca, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/4f961dd0b9744f51a9df3f1c6301a084] to archive 2023-04-28 14:59:26,725 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 14:59:26,726 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/8c095c95c9ef48a3acdd6c8b91e07e4f to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/8c095c95c9ef48a3acdd6c8b91e07e4f 2023-04-28 14:59:26,727 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/ee2438536df941fb984ac1ff91cafc01 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/ee2438536df941fb984ac1ff91cafc01 2023-04-28 14:59:26,728 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/3109db1b49e3434497771ef287f1faa9 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/3109db1b49e3434497771ef287f1faa9 2023-04-28 14:59:26,729 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/b8840ab117f44adaae6ba92611caa7cd to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/b8840ab117f44adaae6ba92611caa7cd 2023-04-28 14:59:26,730 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/b03b201dc1b34eb09a0de3efd6405581 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/b03b201dc1b34eb09a0de3efd6405581 2023-04-28 14:59:26,731 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/37e761bfd59d480eaf40a4350f34914d to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/37e761bfd59d480eaf40a4350f34914d 2023-04-28 14:59:26,732 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/5a9596747a664d78852391077d56fe98 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/5a9596747a664d78852391077d56fe98 2023-04-28 14:59:26,733 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/81a86d93e59c4910b77b25f268bbc5b2 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/81a86d93e59c4910b77b25f268bbc5b2 2023-04-28 14:59:26,734 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/4e8d1b80884b464f81edb7b632c0d0ad to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/4e8d1b80884b464f81edb7b632c0d0ad 2023-04-28 14:59:26,735 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/cb686a39049547249bfcc2530a797c6b to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/cb686a39049547249bfcc2530a797c6b 2023-04-28 14:59:26,736 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/c09d8f4661ba460298d7cb221cd4eb5c to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/c09d8f4661ba460298d7cb221cd4eb5c 2023-04-28 14:59:26,736 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/d6bb6029c1864285bb8fec687db20309 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/d6bb6029c1864285bb8fec687db20309 2023-04-28 14:59:26,737 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/28e6c89bfbec476eb9ae3e404394c0c8 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/28e6c89bfbec476eb9ae3e404394c0c8 2023-04-28 14:59:26,738 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/125223e820e8483db649fa29738d0629 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/125223e820e8483db649fa29738d0629 2023-04-28 14:59:26,739 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/e43699609bc642078717538125e4bfca to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/e43699609bc642078717538125e4bfca 2023-04-28 14:59:26,740 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/79ce9681a6544ef7acdf16e88b47e686 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/79ce9681a6544ef7acdf16e88b47e686 2023-04-28 14:59:26,741 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/22232396f70e4b50ac4639d0ff5ee18e to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/22232396f70e4b50ac4639d0ff5ee18e 2023-04-28 14:59:26,742 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/04699f51638c4662905ff3f1b993c3b5 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/04699f51638c4662905ff3f1b993c3b5 2023-04-28 14:59:26,743 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/48bc3fcf2ff143be9b7eb6ed99d5b72f to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/48bc3fcf2ff143be9b7eb6ed99d5b72f 2023-04-28 14:59:26,744 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/14c439238e5d4c9f8e55658cd1387ef4 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/14c439238e5d4c9f8e55658cd1387ef4 2023-04-28 14:59:26,745 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/3360e845ce3b497280a0e21a3fb06752 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/3360e845ce3b497280a0e21a3fb06752 2023-04-28 14:59:26,746 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/ea93cf5267374e5bbf12d0a6321330ca to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/ea93cf5267374e5bbf12d0a6321330ca 2023-04-28 14:59:26,747 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/4f961dd0b9744f51a9df3f1c6301a084 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/4f961dd0b9744f51a9df3f1c6301a084 2023-04-28 14:59:26,750 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/54c4e7ddbe614602a11ecd19702be32e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/0e2e1d9ec6714bd0beaf28981711369e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/a6a105000c874b78b6788e28ae1ad3c7, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/c874a8d494144edda1cc7ab6c13eb14a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/2195513540f044eeae5cdf4126852a62, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/8849daae8a814f2eb9d8ed2b9dd2286c, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/a76499e561724911af5e02445b5a3348, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/6c316ec39aee4671bc3185a7bab1e54e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/29779d727fa94e2fa235dc344e25fdb2, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/b608dd33a4064230b2c6cab492c6212d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/e02a9df85ab84487a8e12b6d7418bc94, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/e40bf8498de141f5bf6bd51aa1b866c7, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/a0aa5ab2e85646a1ac7cd2a0828c3682, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/129f2a3d2cbe4c269f5c7782d789b646, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/d5e2abb3b780449f9d482e81b4ed1084, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/cc1de880cd7b48f78753cdc817fbcff5, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/4814fda097584cd88c13f66e23acc0af, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/f9f2b396c0d6430daf0f847e8c1aeabe, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/2914d367da704a5f8a0cf07d30d45104, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/292e94dec0104d82ae9cac7e57e8aaef, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/6651d3b7b66444b8b96ee5985ea826c5, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/cf4f505914ff43be854dabf3e38d5c03, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/a34017a0f735434185fc8574bd2ade7e] to archive 2023-04-28 14:59:26,751 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 14:59:26,752 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/54c4e7ddbe614602a11ecd19702be32e to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/54c4e7ddbe614602a11ecd19702be32e 2023-04-28 14:59:26,753 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/0e2e1d9ec6714bd0beaf28981711369e to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/0e2e1d9ec6714bd0beaf28981711369e 2023-04-28 14:59:26,755 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/a6a105000c874b78b6788e28ae1ad3c7 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/a6a105000c874b78b6788e28ae1ad3c7 2023-04-28 14:59:26,756 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/c874a8d494144edda1cc7ab6c13eb14a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/c874a8d494144edda1cc7ab6c13eb14a 2023-04-28 14:59:26,758 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/2195513540f044eeae5cdf4126852a62 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/2195513540f044eeae5cdf4126852a62 2023-04-28 14:59:26,759 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/8849daae8a814f2eb9d8ed2b9dd2286c to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/8849daae8a814f2eb9d8ed2b9dd2286c 2023-04-28 14:59:26,761 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/a76499e561724911af5e02445b5a3348 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/a76499e561724911af5e02445b5a3348 2023-04-28 14:59:26,762 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/6c316ec39aee4671bc3185a7bab1e54e to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/6c316ec39aee4671bc3185a7bab1e54e 2023-04-28 14:59:26,764 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/29779d727fa94e2fa235dc344e25fdb2 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/29779d727fa94e2fa235dc344e25fdb2 2023-04-28 14:59:26,765 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/b608dd33a4064230b2c6cab492c6212d to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/b608dd33a4064230b2c6cab492c6212d 2023-04-28 14:59:26,766 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/e02a9df85ab84487a8e12b6d7418bc94 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/e02a9df85ab84487a8e12b6d7418bc94 2023-04-28 14:59:26,767 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/e40bf8498de141f5bf6bd51aa1b866c7 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/e40bf8498de141f5bf6bd51aa1b866c7 2023-04-28 14:59:26,769 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/a0aa5ab2e85646a1ac7cd2a0828c3682 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/a0aa5ab2e85646a1ac7cd2a0828c3682 2023-04-28 14:59:26,771 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/129f2a3d2cbe4c269f5c7782d789b646 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/129f2a3d2cbe4c269f5c7782d789b646 2023-04-28 14:59:26,772 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/d5e2abb3b780449f9d482e81b4ed1084 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/d5e2abb3b780449f9d482e81b4ed1084 2023-04-28 14:59:26,774 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/cc1de880cd7b48f78753cdc817fbcff5 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/cc1de880cd7b48f78753cdc817fbcff5 2023-04-28 14:59:26,776 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/4814fda097584cd88c13f66e23acc0af to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/4814fda097584cd88c13f66e23acc0af 2023-04-28 14:59:26,778 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/f9f2b396c0d6430daf0f847e8c1aeabe to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/f9f2b396c0d6430daf0f847e8c1aeabe 2023-04-28 14:59:26,780 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/2914d367da704a5f8a0cf07d30d45104 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/2914d367da704a5f8a0cf07d30d45104 2023-04-28 14:59:26,781 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/292e94dec0104d82ae9cac7e57e8aaef to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/292e94dec0104d82ae9cac7e57e8aaef 2023-04-28 14:59:26,783 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/6651d3b7b66444b8b96ee5985ea826c5 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/6651d3b7b66444b8b96ee5985ea826c5 2023-04-28 14:59:26,784 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/cf4f505914ff43be854dabf3e38d5c03 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/cf4f505914ff43be854dabf3e38d5c03 2023-04-28 14:59:26,785 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/a34017a0f735434185fc8574bd2ade7e to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/a34017a0f735434185fc8574bd2ade7e 2023-04-28 14:59:26,788 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/04ac0565409b45fd9c061f1a6eb3d7f7, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/fd212d398664432cbe673de037a969c5, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/679c24c1d188453690150feef68da748, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/2be99a8c1cef412389390bca57fcd486, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/ad2dfd65c80e4f1795202b55afd3a440, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/0612e02eb73344edbc4422dacca62f25, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/144b8d8261864e3280222948f3e20a89, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/46978d3cb8e94a778a67b341d677f992, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/e9ffd2c947bf43cbae8231b08c845aaf, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/37fb6eef6f544122a308ec23dcbfb9e6, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/7b3dc46c66394f26b1ceb860b6d14cbe, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/da83122cb707457aa128a7a0fa89de42, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/383ce69044f049709627266e41ad430a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/b184375536604da3b77fe64d55d91b52, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/f2bbd7903b9f41588811b5ab209d63d9, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/791c3e0cb2564dc39db78ae1fa2315c3, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/9b7062457f1047a99ade9e8eba279f87, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/33bc18aeeae34e67a0bf03958fe064d8, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/696a910367014d49a2ea9567fe3a8373, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/8b88a20deb9c45dd8bf8ecc083048036, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/bfcc09a8f24547c68cd36a55a68c6f28, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/c97c1901e40946ae8d2c3bea31b3f79d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/01975acccd034a4ea0e34a02466c0a8a] to archive 2023-04-28 14:59:26,788 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 14:59:26,790 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/04ac0565409b45fd9c061f1a6eb3d7f7 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/04ac0565409b45fd9c061f1a6eb3d7f7 2023-04-28 14:59:26,791 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/fd212d398664432cbe673de037a969c5 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/fd212d398664432cbe673de037a969c5 2023-04-28 14:59:26,792 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/679c24c1d188453690150feef68da748 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/679c24c1d188453690150feef68da748 2023-04-28 14:59:26,792 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/2be99a8c1cef412389390bca57fcd486 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/2be99a8c1cef412389390bca57fcd486 2023-04-28 14:59:26,793 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/ad2dfd65c80e4f1795202b55afd3a440 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/ad2dfd65c80e4f1795202b55afd3a440 2023-04-28 14:59:26,794 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/0612e02eb73344edbc4422dacca62f25 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/0612e02eb73344edbc4422dacca62f25 2023-04-28 14:59:26,795 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/144b8d8261864e3280222948f3e20a89 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/144b8d8261864e3280222948f3e20a89 2023-04-28 14:59:26,796 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/46978d3cb8e94a778a67b341d677f992 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/46978d3cb8e94a778a67b341d677f992 2023-04-28 14:59:26,797 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/e9ffd2c947bf43cbae8231b08c845aaf to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/e9ffd2c947bf43cbae8231b08c845aaf 2023-04-28 14:59:26,798 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/37fb6eef6f544122a308ec23dcbfb9e6 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/37fb6eef6f544122a308ec23dcbfb9e6 2023-04-28 14:59:26,799 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/7b3dc46c66394f26b1ceb860b6d14cbe to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/7b3dc46c66394f26b1ceb860b6d14cbe 2023-04-28 14:59:26,800 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/da83122cb707457aa128a7a0fa89de42 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/da83122cb707457aa128a7a0fa89de42 2023-04-28 14:59:26,801 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/383ce69044f049709627266e41ad430a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/383ce69044f049709627266e41ad430a 2023-04-28 14:59:26,802 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/b184375536604da3b77fe64d55d91b52 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/b184375536604da3b77fe64d55d91b52 2023-04-28 14:59:26,803 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/f2bbd7903b9f41588811b5ab209d63d9 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/f2bbd7903b9f41588811b5ab209d63d9 2023-04-28 14:59:26,804 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/791c3e0cb2564dc39db78ae1fa2315c3 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/791c3e0cb2564dc39db78ae1fa2315c3 2023-04-28 14:59:26,805 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/9b7062457f1047a99ade9e8eba279f87 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/9b7062457f1047a99ade9e8eba279f87 2023-04-28 14:59:26,806 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/33bc18aeeae34e67a0bf03958fe064d8 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/33bc18aeeae34e67a0bf03958fe064d8 2023-04-28 14:59:26,806 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/696a910367014d49a2ea9567fe3a8373 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/696a910367014d49a2ea9567fe3a8373 2023-04-28 14:59:26,807 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/8b88a20deb9c45dd8bf8ecc083048036 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/8b88a20deb9c45dd8bf8ecc083048036 2023-04-28 14:59:26,808 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/bfcc09a8f24547c68cd36a55a68c6f28 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/bfcc09a8f24547c68cd36a55a68c6f28 2023-04-28 14:59:26,809 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/c97c1901e40946ae8d2c3bea31b3f79d to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/c97c1901e40946ae8d2c3bea31b3f79d 2023-04-28 14:59:26,809 DEBUG [StoreCloser-TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/01975acccd034a4ea0e34a02466c0a8a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/01975acccd034a4ea0e34a02466c0a8a 2023-04-28 14:59:26,812 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/recovered.edits/349.seqid, newMaxSeqId=349, maxSeqId=1 2023-04-28 14:59:26,813 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714. 2023-04-28 14:59:26,813 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1558): Region close journal for 7fd2108ac9c35550edc28d90fafe4714: 2023-04-28 14:59:26,815 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(147): Closed 7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:26,815 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=7fd2108ac9c35550edc28d90fafe4714, regionState=CLOSED 2023-04-28 14:59:26,816 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693966815"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682693966815"}]},"ts":"1682693966815"} 2023-04-28 14:59:26,818 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-28 14:59:26,818 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 7fd2108ac9c35550edc28d90fafe4714, server=jenkins-hbase17.apache.org,37181,1682693812482 in 2.2410 sec 2023-04-28 14:59:26,819 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-28 14:59:26,819 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=7fd2108ac9c35550edc28d90fafe4714, UNASSIGN in 2.2450 sec 2023-04-28 14:59:26,819 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682693966819"}]},"ts":"1682693966819"} 2023-04-28 14:59:26,820 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-28 14:59:26,821 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-28 14:59:26,822 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.2540 sec 2023-04-28 14:59:28,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-28 14:59:28,682 INFO [Listener at localhost.localdomain/40193] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-28 14:59:28,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.HMaster$5(2228): Client=jenkins//136.243.18.41 delete TestAcidGuarantees 2023-04-28 14:59:28,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:59:28,688 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:59:28,689 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:59:28,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-28 14:59:28,698 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:28,703 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A, FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B, FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C, FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/recovered.edits] 2023-04-28 14:59:28,707 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/12ccd5ab53434a2bb7b55cacf9f082b7 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/12ccd5ab53434a2bb7b55cacf9f082b7 2023-04-28 14:59:28,709 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/6513fae5236a46969710ebeeb58c67f3 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/6513fae5236a46969710ebeeb58c67f3 2023-04-28 14:59:28,711 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/78ae3fd7555148d2a4314dc191b373fd to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/A/78ae3fd7555148d2a4314dc191b373fd 2023-04-28 14:59:28,714 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/445d09612222469383b9b2bd066c3a1f to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/445d09612222469383b9b2bd066c3a1f 2023-04-28 14:59:28,716 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/7157c3b1547f4ac8a3b456bb480ee5ed to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/7157c3b1547f4ac8a3b456bb480ee5ed 2023-04-28 14:59:28,717 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/eeff97bfbd9247c3ae8085bfe4221a66 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/B/eeff97bfbd9247c3ae8085bfe4221a66 2023-04-28 14:59:28,720 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/692d5bf315284269a579ad896ec56e29 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/692d5bf315284269a579ad896ec56e29 2023-04-28 14:59:28,722 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/768fd6a750ab4619a8411f89addd65c4 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/768fd6a750ab4619a8411f89addd65c4 2023-04-28 14:59:28,724 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/7d8ca8b71caa45f9945b551a615b2787 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/C/7d8ca8b71caa45f9945b551a615b2787 2023-04-28 14:59:28,728 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/recovered.edits/349.seqid to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714/recovered.edits/349.seqid 2023-04-28 14:59:28,729 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/7fd2108ac9c35550edc28d90fafe4714 2023-04-28 14:59:28,729 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 14:59:28,731 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:59:28,732 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-28 14:59:28,733 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-28 14:59:28,734 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:59:28,735 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-28 14:59:28,735 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682693968735"}]},"ts":"9223372036854775807"} 2023-04-28 14:59:28,736 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-28 14:59:28,736 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 7fd2108ac9c35550edc28d90fafe4714, NAME => 'TestAcidGuarantees,,1682693940433.7fd2108ac9c35550edc28d90fafe4714.', STARTKEY => '', ENDKEY => ''}] 2023-04-28 14:59:28,736 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-28 14:59:28,736 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682693968736"}]},"ts":"9223372036854775807"} 2023-04-28 14:59:28,738 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-28 14:59:28,740 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:59:28,741 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 56 msec 2023-04-28 14:59:28,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-28 14:59:28,791 INFO [Listener at localhost.localdomain/40193] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-28 14:59:28,811 INFO [Listener at localhost.localdomain/40193] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=228 (was 227) - Thread LEAK? -, OpenFileDescriptor=541 (was 529) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=320 (was 334), ProcessCount=168 (was 168), AvailableMemoryMB=5006 (was 4971) - AvailableMemoryMB LEAK? - 2023-04-28 14:59:28,819 INFO [Listener at localhost.localdomain/40193] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228, OpenFileDescriptor=541, MaxFileDescriptor=60000, SystemLoadAverage=320, ProcessCount=168, AvailableMemoryMB=5006 2023-04-28 14:59:28,821 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-28 14:59:28,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.HMaster$4(2112): Client=jenkins//136.243.18.41 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-28 14:59:28,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-28 14:59:28,824 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-28 14:59:28,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(697): Client=jenkins//136.243.18.41 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-28 14:59:28,824 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-28 14:59:28,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-28 14:59:28,825 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:28,826 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13 empty. 2023-04-28 14:59:28,826 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:28,826 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 14:59:28,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-28 14:59:29,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-28 14:59:29,233 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-28 14:59:29,234 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 2e61d8a0a5c8be60fbc343435192fa13, NAME => 'TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp 2023-04-28 14:59:29,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-28 14:59:29,641 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 14:59:29,641 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 2e61d8a0a5c8be60fbc343435192fa13, disabling compactions & flushes 2023-04-28 14:59:29,642 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:29,642 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:29,642 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. after waiting 0 ms 2023-04-28 14:59:29,642 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:29,642 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:29,642 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:29,648 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-28 14:59:29,650 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693969650"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682693969650"}]},"ts":"1682693969650"} 2023-04-28 14:59:29,652 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-28 14:59:29,653 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-28 14:59:29,654 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682693969653"}]},"ts":"1682693969653"} 2023-04-28 14:59:29,655 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-28 14:59:29,660 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2e61d8a0a5c8be60fbc343435192fa13, ASSIGN}] 2023-04-28 14:59:29,662 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2e61d8a0a5c8be60fbc343435192fa13, ASSIGN 2023-04-28 14:59:29,663 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=2e61d8a0a5c8be60fbc343435192fa13, ASSIGN; state=OFFLINE, location=jenkins-hbase17.apache.org,37181,1682693812482; forceNewPlan=false, retain=false 2023-04-28 14:59:29,815 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=2e61d8a0a5c8be60fbc343435192fa13, regionState=OPENING, regionLocation=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:29,816 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693969815"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682693969815"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682693969815"}]},"ts":"1682693969815"} 2023-04-28 14:59:29,819 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482}] 2023-04-28 14:59:29,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-28 14:59:29,984 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:29,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2e61d8a0a5c8be60fbc343435192fa13, NAME => 'TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.', STARTKEY => '', ENDKEY => ''} 2023-04-28 14:59:29,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:29,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 14:59:29,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7894): checking encryption for 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:29,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7897): checking classloading for 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:29,988 INFO [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:29,991 DEBUG [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A 2023-04-28 14:59:29,991 DEBUG [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A 2023-04-28 14:59:29,991 INFO [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 14:59:29,991 INFO [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2e61d8a0a5c8be60fbc343435192fa13 columnFamilyName A 2023-04-28 14:59:29,992 INFO [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] regionserver.HStore(310): Store=2e61d8a0a5c8be60fbc343435192fa13/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:59:29,992 INFO [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:29,994 DEBUG [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B 2023-04-28 14:59:29,995 DEBUG [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B 2023-04-28 14:59:29,995 INFO [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 14:59:29,995 INFO [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2e61d8a0a5c8be60fbc343435192fa13 columnFamilyName B 2023-04-28 14:59:29,996 INFO [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] regionserver.HStore(310): Store=2e61d8a0a5c8be60fbc343435192fa13/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:59:29,996 INFO [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:29,998 DEBUG [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C 2023-04-28 14:59:29,998 DEBUG [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C 2023-04-28 14:59:29,998 INFO [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 14:59:29,998 INFO [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2e61d8a0a5c8be60fbc343435192fa13 columnFamilyName C 2023-04-28 14:59:29,999 INFO [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] regionserver.HStore(310): Store=2e61d8a0a5c8be60fbc343435192fa13/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:59:29,999 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:30,000 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:30,001 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:30,004 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 14:59:30,005 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1055): writing seq id for 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:30,007 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-28 14:59:30,007 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1072): Opened 2e61d8a0a5c8be60fbc343435192fa13; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67449377, jitterRate=0.005074039101600647}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 14:59:30,008 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(965): Region open journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:30,008 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13., pid=56, masterSystemTime=1682693969972 2023-04-28 14:59:30,009 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:30,009 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:30,009 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=2e61d8a0a5c8be60fbc343435192fa13, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:30,009 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693970009"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1682693970009"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682693970009"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682693970009"}]},"ts":"1682693970009"} 2023-04-28 14:59:30,011 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-28 14:59:30,011 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 in 191 msec 2023-04-28 14:59:30,012 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-28 14:59:30,012 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2e61d8a0a5c8be60fbc343435192fa13, ASSIGN in 352 msec 2023-04-28 14:59:30,013 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-28 14:59:30,013 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682693970013"}]},"ts":"1682693970013"} 2023-04-28 14:59:30,013 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-28 14:59:30,015 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-28 14:59:30,016 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1930 sec 2023-04-28 14:59:30,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-28 14:59:30,934 INFO [Listener at localhost.localdomain/40193] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-28 14:59:30,935 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x50131d14 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:59:30,943 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5be55982, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:59:30,945 DEBUG [Listener at localhost.localdomain/40193] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:59:30,947 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:39914, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:59:30,949 DEBUG [Listener at localhost.localdomain/40193] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-28 14:59:30,952 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:37902, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-28 14:59:30,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-28 14:59:30,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.HMaster$12(2454): Client=jenkins//136.243.18.41 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-28 14:59:30,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-28 14:59:31,369 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-28 14:59:31,370 DEBUG [PEWorker-5] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-28 14:59:31,371 INFO [PEWorker-5] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-28 14:59:31,374 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-28 14:59:31,378 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2e61d8a0a5c8be60fbc343435192fa13, REOPEN/MOVE}] 2023-04-28 14:59:31,379 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2e61d8a0a5c8be60fbc343435192fa13, REOPEN/MOVE 2023-04-28 14:59:31,380 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=2e61d8a0a5c8be60fbc343435192fa13, regionState=CLOSING, regionLocation=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:31,380 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693971380"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682693971380"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682693971380"}]},"ts":"1682693971380"} 2023-04-28 14:59:31,382 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482}] 2023-04-28 14:59:31,534 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(109): Close 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:31,536 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1604): Closing 2e61d8a0a5c8be60fbc343435192fa13, disabling compactions & flushes 2023-04-28 14:59:31,536 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:31,536 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:31,536 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. after waiting 0 ms 2023-04-28 14:59:31,536 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:31,544 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-28 14:59:31,545 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:31,545 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1558): Region close journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:31,545 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 2e61d8a0a5c8be60fbc343435192fa13 to self. 2023-04-28 14:59:31,547 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(147): Closed 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:31,548 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=2e61d8a0a5c8be60fbc343435192fa13, regionState=CLOSED 2023-04-28 14:59:31,548 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693971548"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682693971548"}]},"ts":"1682693971548"} 2023-04-28 14:59:31,552 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-28 14:59:31,552 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 in 167 msec 2023-04-28 14:59:31,553 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=2e61d8a0a5c8be60fbc343435192fa13, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase17.apache.org,37181,1682693812482; forceNewPlan=false, retain=true 2023-04-28 14:59:31,703 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=2e61d8a0a5c8be60fbc343435192fa13, regionState=OPENING, regionLocation=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:31,703 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693971703"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682693971703"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682693971703"}]},"ts":"1682693971703"} 2023-04-28 14:59:31,706 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482}] 2023-04-28 14:59:31,861 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:31,861 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2e61d8a0a5c8be60fbc343435192fa13, NAME => 'TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.', STARTKEY => '', ENDKEY => ''} 2023-04-28 14:59:31,861 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:31,861 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 14:59:31,862 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7894): checking encryption for 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:31,862 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7897): checking classloading for 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:31,863 INFO [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:31,863 DEBUG [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A 2023-04-28 14:59:31,863 DEBUG [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A 2023-04-28 14:59:31,863 INFO [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 14:59:31,864 INFO [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2e61d8a0a5c8be60fbc343435192fa13 columnFamilyName A 2023-04-28 14:59:31,865 INFO [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] regionserver.HStore(310): Store=2e61d8a0a5c8be60fbc343435192fa13/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:59:31,865 INFO [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:31,866 DEBUG [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B 2023-04-28 14:59:31,866 DEBUG [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B 2023-04-28 14:59:31,866 INFO [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 14:59:31,866 INFO [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2e61d8a0a5c8be60fbc343435192fa13 columnFamilyName B 2023-04-28 14:59:31,866 INFO [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] regionserver.HStore(310): Store=2e61d8a0a5c8be60fbc343435192fa13/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:59:31,867 INFO [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:31,867 DEBUG [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C 2023-04-28 14:59:31,867 DEBUG [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C 2023-04-28 14:59:31,867 INFO [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 14:59:31,868 INFO [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2e61d8a0a5c8be60fbc343435192fa13 columnFamilyName C 2023-04-28 14:59:31,868 INFO [StoreOpener-2e61d8a0a5c8be60fbc343435192fa13-1] regionserver.HStore(310): Store=2e61d8a0a5c8be60fbc343435192fa13/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 14:59:31,868 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:31,869 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:31,870 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:31,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 14:59:31,872 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1055): writing seq id for 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:31,872 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1072): Opened 2e61d8a0a5c8be60fbc343435192fa13; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65627944, jitterRate=-0.02206742763519287}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 14:59:31,872 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(965): Region open journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:31,873 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13., pid=61, masterSystemTime=1682693971859 2023-04-28 14:59:31,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:31,874 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:31,874 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=2e61d8a0a5c8be60fbc343435192fa13, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:31,874 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693971874"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1682693971874"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682693971874"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682693971874"}]},"ts":"1682693971874"} 2023-04-28 14:59:31,876 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-28 14:59:31,876 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 in 169 msec 2023-04-28 14:59:31,877 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-28 14:59:31,877 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2e61d8a0a5c8be60fbc343435192fa13, REOPEN/MOVE in 498 msec 2023-04-28 14:59:31,878 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-28 14:59:31,879 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 503 msec 2023-04-28 14:59:31,880 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 924 msec 2023-04-28 14:59:31,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-28 14:59:31,881 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x6722b207 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:59:31,885 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2f11e84a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:59:31,885 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x48502121 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:59:31,888 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3a1010a6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:59:31,889 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x08d3d8bc to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:59:31,892 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4143b9ae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:59:31,892 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x7e6ad437 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:59:31,895 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@686af57e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:59:31,896 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x2f6f7187 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:59:31,900 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@78cd331, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:59:31,901 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x1d74fc48 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:59:31,906 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7c00a18c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:59:31,906 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x070ab5f5 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:59:31,911 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3da68fe0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:59:31,911 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x2daeb795 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:59:31,917 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7fd90ac8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:59:31,917 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x28ac2b09 to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:59:31,925 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70e63396, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:59:31,925 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(139): Connect 0x7bf1c01c to 127.0.0.1:61762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 14:59:31,929 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@32cc4808, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 14:59:31,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:31,935 DEBUG [hconnection-0x25dfdfb7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:59:31,936 DEBUG [hconnection-0x3bc4d424-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:59:31,936 DEBUG [hconnection-0x6cb6e05-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:59:31,937 DEBUG [hconnection-0x1a97d513-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:59:31,938 DEBUG [hconnection-0x15d6065f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:59:31,938 DEBUG [hconnection-0x9c2e566-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:59:31,938 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:39948, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:59:31,938 DEBUG [hconnection-0x6701fd88-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:59:31,939 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:39960, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:59:31,941 DEBUG [hconnection-0x7a3a103b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:59:31,942 DEBUG [hconnection-0x61728eca-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:59:31,943 DEBUG [hconnection-0x1a66e378-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 14:59:31,947 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:39932, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:59:31,947 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:39994, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:59:31,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:31,947 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:39920, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:59:31,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:31,947 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:39958, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:59:31,947 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:39954, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:59:31,947 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:31,947 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:39988, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:59:31,949 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:39972, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:59:31,948 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:31,950 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:31,951 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:31,958 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:40006, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 14:59:31,958 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:31,958 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:31,959 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:31,959 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:31,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:31,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:31,960 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:31,960 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:31,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:31,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:31,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:31,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:31,961 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:31,961 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:31,961 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:31,961 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:31,961 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:31,961 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:31,962 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:31,962 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:31,962 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:31,962 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:31,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:31,963 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:31,963 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2e61d8a0a5c8be60fbc343435192fa13 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 14:59:31,980 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=A 2023-04-28 14:59:31,980 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:31,980 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=B 2023-04-28 14:59:31,980 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:31,980 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=C 2023-04-28 14:59:31,980 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:32,011 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:32,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39988 deadline: 1682694032009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:32,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693972006, max timestamp=1682693972011Num uniques 859; 2023-04-28 14:59:32,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:59:32,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693972006, max timestamp=1682693972011Num uniques 859; 2023-04-28 14:59:32,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:59:32,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693972006, max timestamp=1682693972011Num uniques 859; 2023-04-28 14:59:32,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:59:32,023 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:32,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:40006 deadline: 1682694032021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:32,023 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:32,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39972 deadline: 1682694032021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:32,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:32,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39948 deadline: 1682694032023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:32,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:32,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694032023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:32,114 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:32,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39988 deadline: 1682694032112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:32,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:32,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:40006 deadline: 1682694032124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:32,126 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:32,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39972 deadline: 1682694032124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:32,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:32,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39948 deadline: 1682694032125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:32,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:32,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694032126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:32,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:32,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39988 deadline: 1682694032316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:32,328 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:32,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39972 deadline: 1682694032327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:32,329 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-28 14:59:32,329 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:32,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:40006 deadline: 1682694032327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:32,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:32,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39948 deadline: 1682694032328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:32,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:32,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694032329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:32,390 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428fb56ccae7a6445c799f43c85957f322c to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428fb56ccae7a6445c799f43c85957f322c 2023-04-28 14:59:32,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:32,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39988 deadline: 1682694032617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:32,631 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:32,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39972 deadline: 1682694032630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:32,634 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:32,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39948 deadline: 1682694032632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:32,634 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:32,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:40006 deadline: 1682694032632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:32,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:32,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694032634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:32,794 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/d33570364e1f44098f94b6de805f1685 2023-04-28 14:59:33,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:33,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39988 deadline: 1682694033122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:33,135 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:33,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39972 deadline: 1682694033134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:33,136 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:33,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39948 deadline: 1682694033134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:33,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:33,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694033138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:33,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:33,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:40006 deadline: 1682694033139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:33,201 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/bc826cce2a9148969757e78d4333ace2 2023-04-28 14:59:33,209 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/a2f36f9f09344f638c948bfe5ca77a1f 2023-04-28 14:59:33,212 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/d33570364e1f44098f94b6de805f1685 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/d33570364e1f44098f94b6de805f1685 2023-04-28 14:59:33,215 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/d33570364e1f44098f94b6de805f1685, entries=100, sequenceid=16, filesize=18.4 K 2023-04-28 14:59:33,215 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/bc826cce2a9148969757e78d4333ace2 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/bc826cce2a9148969757e78d4333ace2 2023-04-28 14:59:33,218 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/bc826cce2a9148969757e78d4333ace2, entries=100, sequenceid=16, filesize=9.3 K 2023-04-28 14:59:33,218 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/a2f36f9f09344f638c948bfe5ca77a1f as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/a2f36f9f09344f638c948bfe5ca77a1f 2023-04-28 14:59:33,221 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/a2f36f9f09344f638c948bfe5ca77a1f, entries=100, sequenceid=16, filesize=9.3 K 2023-04-28 14:59:33,221 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.47 KB/130530 for 2e61d8a0a5c8be60fbc343435192fa13 in 1258ms, sequenceid=16, compaction requested=false 2023-04-28 14:59:33,221 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-28 14:59:33,222 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:33,222 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:33,222 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:33,222 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:33,222 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:33,222 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:33,222 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:33,223 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:33,223 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:33,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:33,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:33,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:33,223 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:33,223 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:33,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:33,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:33,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:33,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:33,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:33,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:33,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6801aac8[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:33,225 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:33,225 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:33,225 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:33,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:33,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:33,225 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:33,225 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:33,225 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:33,225 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:33,225 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:33,226 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:33,226 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:33,226 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:33,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:33,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:33,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:33,226 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:33,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:33,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:33,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:33,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:33,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:33,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:33,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:33,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:33,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:33,228 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:33,228 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:33,228 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:33,228 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:33,228 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:33,229 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:33,229 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:33,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:33,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:33,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:33,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:33,229 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:33,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:33,230 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:33,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:33,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:33,231 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:33,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:33,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:33,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:33,233 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:33,233 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:33,233 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:33,233 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:33,233 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:33,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:33,233 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:33,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:33,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:33,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:33,233 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:33,233 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:33,233 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:33,233 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:33,233 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:33,233 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:33,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:33,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:33,333 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:33,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:33,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:33,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:33,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:33,336 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:33,336 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:33,336 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:33,336 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:33,336 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:33,337 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:33,337 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:33,337 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:33,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:33,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:33,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:33,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:33,338 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:33,338 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:33,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:33,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:33,338 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:33,338 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:33,339 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:33,339 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:33,339 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:33,339 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:33,339 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:33,339 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:33,339 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:33,339 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:33,339 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2e61d8a0a5c8be60fbc343435192fa13 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-28 14:59:33,339 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=A 2023-04-28 14:59:33,339 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:59:33,339 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=B 2023-04-28 14:59:33,340 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:59:33,340 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=C 2023-04-28 14:59:33,340 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:59:33,752 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428a380e730c8104b13a99c53d1daae570d to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a380e730c8104b13a99c53d1daae570d 2023-04-28 14:59:34,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:34,131 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:34,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:34,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39988 deadline: 1682694034142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:34,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:34,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694034144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:34,147 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:34,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39948 deadline: 1682694034144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:34,147 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:34,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39972 deadline: 1682694034145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:34,147 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:34,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:40006 deadline: 1682694034145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:34,156 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/1cfb20a90be9401da2b6b159372f44bd 2023-04-28 14:59:34,175 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/0448540e08dc45daba93ccb7c0bd3177 2023-04-28 14:59:34,247 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:34,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39988 deadline: 1682694034246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:34,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:34,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39972 deadline: 1682694034248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:34,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:34,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39988 deadline: 1682694034448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:34,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:34,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39972 deadline: 1682694034451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:34,583 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/0d4c0e9b0f614b08a35a3e238e6ce331 2023-04-28 14:59:34,586 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/1cfb20a90be9401da2b6b159372f44bd as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/1cfb20a90be9401da2b6b159372f44bd 2023-04-28 14:59:34,589 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/1cfb20a90be9401da2b6b159372f44bd, entries=150, sequenceid=51, filesize=25.1 K 2023-04-28 14:59:34,589 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/0448540e08dc45daba93ccb7c0bd3177 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/0448540e08dc45daba93ccb7c0bd3177 2023-04-28 14:59:34,592 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/0448540e08dc45daba93ccb7c0bd3177, entries=150, sequenceid=51, filesize=11.6 K 2023-04-28 14:59:34,592 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/0d4c0e9b0f614b08a35a3e238e6ce331 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/0d4c0e9b0f614b08a35a3e238e6ce331 2023-04-28 14:59:34,595 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/0d4c0e9b0f614b08a35a3e238e6ce331, entries=150, sequenceid=51, filesize=11.6 K 2023-04-28 14:59:34,595 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 2e61d8a0a5c8be60fbc343435192fa13 in 1256ms, sequenceid=51, compaction requested=false 2023-04-28 14:59:34,596 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:34,596 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:34,596 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:34,596 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:34,596 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:34,596 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:34,596 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:34,597 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:34,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:34,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:34,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:34,597 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:34,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:34,597 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:34,597 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:34,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:34,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:34,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:34,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:34,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:34,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dd1011[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:34,598 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:34,598 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:34,598 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:34,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:34,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:34,599 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:34,599 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:34,599 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:34,599 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:34,599 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:34,599 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:34,599 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:34,599 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:34,599 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:34,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:34,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:34,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:34,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:34,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:34,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:34,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:34,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:34,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:34,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:34,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:34,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:34,602 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:34,602 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:34,602 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:34,602 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:34,602 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:34,603 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:34,603 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:34,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:34,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:34,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:34,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:34,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:34,603 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:34,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:34,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:34,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:34,603 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:34,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:34,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:34,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:34,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:34,604 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:34,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:34,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:34,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:34,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:34,606 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:34,606 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:34,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:34,606 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:34,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:34,606 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:34,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:34,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:34,606 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:34,607 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:34,607 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:34,607 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:34,607 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:34,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:34,607 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:34,607 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:34,607 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:34,707 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:34,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:34,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:34,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:34,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:34,709 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:34,709 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:34,709 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:34,709 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:34,710 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:34,710 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:34,710 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:34,710 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:34,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:34,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:34,711 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:34,711 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:34,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:34,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:34,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:34,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:34,711 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:34,711 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:34,711 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:34,711 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:34,711 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:34,712 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:34,712 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:34,712 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:34,712 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:34,712 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2e61d8a0a5c8be60fbc343435192fa13 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-28 14:59:34,712 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=A 2023-04-28 14:59:34,712 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:34,712 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=B 2023-04-28 14:59:34,712 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:34,712 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=C 2023-04-28 14:59:34,712 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:34,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:34,752 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:34,776 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693974769, max timestamp=1682693974775Num uniques 859; 2023-04-28 14:59:34,776 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:59:34,776 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693974770, max timestamp=1682693974775Num uniques 859; 2023-04-28 14:59:34,776 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:59:34,776 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693974769, max timestamp=1682693974775Num uniques 859; 2023-04-28 14:59:34,776 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:59:34,797 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:34,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39972 deadline: 1682694034796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:34,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:34,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39988 deadline: 1682694034797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:34,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:34,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39972 deadline: 1682694034898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:34,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:34,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39988 deadline: 1682694034900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:35,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:35,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39972 deadline: 1682694035102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:35,106 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:35,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39988 deadline: 1682694035105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:35,121 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304284211cdd39a8f432887dee0b4f2bf61ce to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304284211cdd39a8f432887dee0b4f2bf61ce 2023-04-28 14:59:35,125 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=62, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/135518df887a4af1a615d8456c5f95fc 2023-04-28 14:59:35,409 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:35,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39972 deadline: 1682694035406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:35,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:35,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39988 deadline: 1682694035409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:35,533 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/edd253c343ee48c7b483a50cd7dfb105 2023-04-28 14:59:35,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:35,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39988 deadline: 1682694035911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:35,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:35,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39972 deadline: 1682694035912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:35,940 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/6530f5243efe43e8ab7b410c8d2cfe99 2023-04-28 14:59:35,943 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/135518df887a4af1a615d8456c5f95fc as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/135518df887a4af1a615d8456c5f95fc 2023-04-28 14:59:35,946 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/135518df887a4af1a615d8456c5f95fc, entries=150, sequenceid=62, filesize=25.1 K 2023-04-28 14:59:35,947 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/edd253c343ee48c7b483a50cd7dfb105 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/edd253c343ee48c7b483a50cd7dfb105 2023-04-28 14:59:35,950 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/edd253c343ee48c7b483a50cd7dfb105, entries=150, sequenceid=62, filesize=11.6 K 2023-04-28 14:59:35,951 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/6530f5243efe43e8ab7b410c8d2cfe99 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/6530f5243efe43e8ab7b410c8d2cfe99 2023-04-28 14:59:35,953 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/6530f5243efe43e8ab7b410c8d2cfe99, entries=150, sequenceid=62, filesize=11.6 K 2023-04-28 14:59:35,954 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=125.23 KB/128240 for 2e61d8a0a5c8be60fbc343435192fa13 in 1242ms, sequenceid=62, compaction requested=true 2023-04-28 14:59:35,954 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:35,954 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:35,954 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:35,954 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:35,955 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:35,955 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:35,955 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:35,957 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:35,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:35,957 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:35,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:35,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:35,957 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:35,957 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:35,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:35,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:35,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:35,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:35,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:35,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:35,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b413cd2[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:35,958 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:35,958 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:35,959 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:35,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:35,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:35,959 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:35,959 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:35,959 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:35,959 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:35,959 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:35,960 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:35,960 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:35,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:35,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:35,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:35,960 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:35,960 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:35,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:35,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:35,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:35,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:35,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:35,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:35,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:35,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:35,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:35,962 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:35,962 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:35,962 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:35,962 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:35,962 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:35,963 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:35,963 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:35,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:35,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:35,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:35,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:35,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:35,963 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:35,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:35,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:35,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:35,964 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:35,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:35,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:35,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:35,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:35,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:35,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:35,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:35,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:35,966 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:35,967 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:35,967 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:35,967 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:35,967 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:35,967 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:35,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:35,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:35,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:35,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:35,967 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:35,967 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:35,967 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:35,968 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:35,968 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:35,968 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:35,968 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:35,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:35,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:36,068 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:36,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:36,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:36,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:36,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:36,070 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:36,071 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:36,071 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:36,071 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:36,071 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:36,071 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:36,071 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:36,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:36,072 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:36,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:36,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:36,072 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:36,072 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:36,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:36,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:36,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:36,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:36,073 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:36,073 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:36,073 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:36,073 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:36,073 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:36,073 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:36,073 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:36,073 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:36,073 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:36,073 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2e61d8a0a5c8be60fbc343435192fa13 3/3 column families, dataSize=125.23 KB heapSize=355.30 KB 2023-04-28 14:59:36,074 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=A 2023-04-28 14:59:36,074 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:59:36,074 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=B 2023-04-28 14:59:36,074 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:59:36,074 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=C 2023-04-28 14:59:36,074 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:59:36,084 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428c3c65d94e5e9492ea16e858855f9eaee to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428c3c65d94e5e9492ea16e858855f9eaee 2023-04-28 14:59:36,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:36,149 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:36,164 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:36,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39948 deadline: 1682694036162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:36,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:36,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694036163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:36,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:36,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:40006 deadline: 1682694036163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:36,267 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:36,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39948 deadline: 1682694036265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:36,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:36,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694036266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:36,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:36,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:40006 deadline: 1682694036266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:36,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:36,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39948 deadline: 1682694036468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:36,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:36,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:40006 deadline: 1682694036468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:36,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:36,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694036469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:36,488 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=97, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/4a9a539bd77140bb83b19bccfcab251f 2023-04-28 14:59:36,499 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/ff4a96be2f234f15ab0158baf42d012a 2023-04-28 14:59:36,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:36,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39948 deadline: 1682694036771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:36,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:36,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694036772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:36,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:36,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:40006 deadline: 1682694036772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:36,907 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/3fbc2dc6e9444cf0805caa67dee50bd2 2023-04-28 14:59:36,910 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/4a9a539bd77140bb83b19bccfcab251f as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/4a9a539bd77140bb83b19bccfcab251f 2023-04-28 14:59:36,915 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/4a9a539bd77140bb83b19bccfcab251f, entries=150, sequenceid=97, filesize=25.1 K 2023-04-28 14:59:36,915 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/ff4a96be2f234f15ab0158baf42d012a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/ff4a96be2f234f15ab0158baf42d012a 2023-04-28 14:59:36,919 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/ff4a96be2f234f15ab0158baf42d012a, entries=150, sequenceid=97, filesize=11.6 K 2023-04-28 14:59:36,919 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/3fbc2dc6e9444cf0805caa67dee50bd2 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/3fbc2dc6e9444cf0805caa67dee50bd2 2023-04-28 14:59:36,923 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/3fbc2dc6e9444cf0805caa67dee50bd2, entries=150, sequenceid=97, filesize=11.6 K 2023-04-28 14:59:36,924 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~125.23 KB/128240, heapSize ~355.25 KB/363776, currentSize=53.67 KB/54960 for 2e61d8a0a5c8be60fbc343435192fa13 in 851ms, sequenceid=97, compaction requested=true 2023-04-28 14:59:36,924 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:36,924 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:36,924 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:36,924 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:36,924 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:36,926 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:36,926 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:36,927 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:36,927 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:36,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:36,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:36,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:36,927 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:36,927 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:36,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:36,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:36,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:36,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:36,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:36,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:36,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63afc4ea[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:36,928 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:36,928 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:36,929 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:36,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:36,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:36,929 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:36,929 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:36,929 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:36,929 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:36,929 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:36,930 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:36,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:36,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:36,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:36,930 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:36,930 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:36,930 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:36,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:36,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:36,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:36,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:36,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:36,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:36,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:36,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:36,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:36,932 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:36,932 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:36,932 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:36,932 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:36,932 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:36,933 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:36,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:36,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:36,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:36,933 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:36,933 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:36,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:36,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:36,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:36,934 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:36,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:36,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:36,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:36,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:36,934 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2e61d8a0a5c8be60fbc343435192fa13 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-28 14:59:36,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:36,934 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=A 2023-04-28 14:59:36,934 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:36,934 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=B 2023-04-28 14:59:36,934 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:36,935 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=C 2023-04-28 14:59:36,935 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:36,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:36,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:36,935 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:36,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:36,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:36,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:36,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:36,937 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:36,937 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:36,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:36,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:36,937 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:36,937 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:36,937 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:36,937 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:36,937 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:36,937 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:36,937 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:36,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:36,937 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:36,937 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:36,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:36,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:36,937 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:36,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:36,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:36,944 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304287c38d8b2ace547c8ba449259ef7ecbf3 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304287c38d8b2ace547c8ba449259ef7ecbf3 2023-04-28 14:59:36,948 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=109, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/6654e74727b64370acbfbb6acff70ff2 2023-04-28 14:59:36,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693976960, max timestamp=1682693976973Num uniques 859; 2023-04-28 14:59:36,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:59:36,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693976960, max timestamp=1682693976973Num uniques 859; 2023-04-28 14:59:36,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:59:36,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693976960, max timestamp=1682693976973Num uniques 859; 2023-04-28 14:59:36,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:59:37,002 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:37,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39988 deadline: 1682694037000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:37,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39972 deadline: 1682694037001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,037 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:37,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:37,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:37,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:37,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:37,040 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:37,040 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:37,040 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:37,040 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:37,040 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:37,040 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:37,041 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,041 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:37,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:37,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:37,041 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,041 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:37,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:37,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:37,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:37,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:37,042 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:37,042 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:37,042 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:37,042 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:37,042 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:37,042 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:37,042 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:37,042 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:37,042 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:37,042 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:37,042 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:37,042 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:37,042 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:37,042 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:37,042 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,042 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:37,043 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:37,043 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,043 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:37,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:37,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:37,044 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:37,044 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:37,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:37,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:37,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:37,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:37,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:37,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61d75bd[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:37,045 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:37,045 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:37,045 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:37,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:37,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:37,045 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:37,045 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:37,045 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:37,045 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,045 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:37,046 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:37,046 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:37,046 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,046 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:37,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:37,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:37,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:37,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:37,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:37,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:37,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:37,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:37,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:37,048 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:37,048 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:37,048 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:37,048 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:37,048 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:37,049 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:37,049 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:37,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:37,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:37,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:37,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:37,049 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:37,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:37,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:37,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:37,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:37,049 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:37,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:37,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:37,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:37,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,050 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:37,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:37,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,052 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,052 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:37,052 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:37,052 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,052 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:37,052 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:37,052 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:37,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:37,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:37,052 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:37,052 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:37,052 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:37,052 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:37,052 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:37,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:37,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:37,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:37,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:37,105 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:37,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39988 deadline: 1682694037103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,105 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:37,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39972 deadline: 1682694037103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,152 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:37,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:37,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:37,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:37,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:37,155 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:37,155 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:37,155 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:37,155 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:37,155 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:37,155 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:37,156 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,156 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:37,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:37,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:37,156 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,156 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:37,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:37,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:37,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:37,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:37,157 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:37,157 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:37,157 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:37,157 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:37,157 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:37,157 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:37,157 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:37,157 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:37,157 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:37,157 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:37,157 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:37,157 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:37,157 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:37,157 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:37,158 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,158 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:37,158 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:37,158 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,158 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:37,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:37,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:37,159 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:37,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:37,159 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:37,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:37,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:37,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:37,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:37,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69afa0f7[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:37,160 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:37,160 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:37,160 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:37,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:37,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:37,160 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:37,160 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:37,160 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:37,160 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,160 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:37,162 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:37,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:37,162 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:37,162 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:37,162 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:37,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:37,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:37,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:37,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:37,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:37,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:37,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:37,163 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:37,163 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:37,163 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:37,163 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:37,163 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:37,164 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:37,164 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:37,164 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:37,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:37,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:37,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:37,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:37,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:37,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:37,165 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:37,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:37,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:37,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:37,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:37,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:37,166 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:37,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:37,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,167 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,167 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:37,167 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:37,167 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:37,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:37,167 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:37,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:37,167 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:37,167 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:37,167 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:37,167 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:37,167 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:37,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:37,167 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:37,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:37,167 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:37,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:37,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:37,268 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:37,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:37,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:37,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:37,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:37,271 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:37,271 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:37,271 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:37,271 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:37,271 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:37,271 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:37,272 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,272 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:37,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:37,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:37,272 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,272 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:37,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:37,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:37,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:37,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:37,273 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:37,273 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:37,273 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:37,273 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:37,273 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:37,273 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:37,273 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:37,273 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:37,273 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:37,273 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:37,273 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:37,273 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:37,273 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:37,273 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:37,273 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,273 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:37,274 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:37,274 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,274 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:37,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:37,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:37,274 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:37,275 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:37,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:37,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:37,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:37,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:37,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:37,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b7aa6bc[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:37,276 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:37,276 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:37,276 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:37,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:37,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:37,276 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:37,276 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:37,276 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:37,276 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,276 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:37,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:37,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39948 deadline: 1682694037275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,277 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,277 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:37,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:37,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:37,277 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:37,277 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:37,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:37,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:37,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:37,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:37,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:37,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:37,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:37,279 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:37,279 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:37,279 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:37,279 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:37,279 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:37,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:37,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694037277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:37,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:40006 deadline: 1682694037279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,280 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:37,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:37,280 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:37,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:37,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:37,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:37,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:37,280 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:37,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:37,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:37,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:37,280 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:37,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:37,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:37,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:37,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,281 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:37,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:37,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,283 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,283 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:37,283 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:37,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:37,283 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:37,283 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:37,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:37,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:37,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:37,283 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:37,283 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,283 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:37,283 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:37,283 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:37,283 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:37,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:37,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:37,284 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:37,308 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:37,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39988 deadline: 1682694037306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:37,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39972 deadline: 1682694037307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,357 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/3ac1cf8e19914527af60193d0bf6d440 2023-04-28 14:59:37,363 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-28 14:59:37,384 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:37,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:37,386 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/a7ff446eefe4498485e64b65be77168e 2023-04-28 14:59:37,391 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/6654e74727b64370acbfbb6acff70ff2 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/6654e74727b64370acbfbb6acff70ff2 2023-04-28 14:59:37,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:37,395 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/6654e74727b64370acbfbb6acff70ff2, entries=150, sequenceid=109, filesize=25.1 K 2023-04-28 14:59:37,396 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/3ac1cf8e19914527af60193d0bf6d440 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/3ac1cf8e19914527af60193d0bf6d440 2023-04-28 14:59:37,399 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/3ac1cf8e19914527af60193d0bf6d440, entries=100, sequenceid=109, filesize=9.3 K 2023-04-28 14:59:37,400 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/a7ff446eefe4498485e64b65be77168e as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/a7ff446eefe4498485e64b65be77168e 2023-04-28 14:59:37,403 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/a7ff446eefe4498485e64b65be77168e, entries=100, sequenceid=109, filesize=9.3 K 2023-04-28 14:59:37,404 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 2e61d8a0a5c8be60fbc343435192fa13 in 470ms, sequenceid=109, compaction requested=true 2023-04-28 14:59:37,404 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:37,404 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:37,404 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 14:59:37,404 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 14:59:37,404 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:37,405 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:37,405 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 54742 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 14:59:37,406 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 2e61d8a0a5c8be60fbc343435192fa13/B is initiating minor compaction (all files) 2023-04-28 14:59:37,406 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e61d8a0a5c8be60fbc343435192fa13/B in TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:37,406 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/bc826cce2a9148969757e78d4333ace2, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/0448540e08dc45daba93ccb7c0bd3177, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/edd253c343ee48c7b483a50cd7dfb105, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/ff4a96be2f234f15ab0158baf42d012a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/3ac1cf8e19914527af60193d0bf6d440] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp, totalSize=53.5 K 2023-04-28 14:59:37,407 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting bc826cce2a9148969757e78d4333ace2, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682693971961 2023-04-28 14:59:37,407 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 121778 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 14:59:37,407 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): 2e61d8a0a5c8be60fbc343435192fa13/A is initiating minor compaction (all files) 2023-04-28 14:59:37,407 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e61d8a0a5c8be60fbc343435192fa13/A in TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:37,407 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 0448540e08dc45daba93ccb7c0bd3177, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682693972020 2023-04-28 14:59:37,407 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/d33570364e1f44098f94b6de805f1685, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/1cfb20a90be9401da2b6b159372f44bd, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/135518df887a4af1a615d8456c5f95fc, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/4a9a539bd77140bb83b19bccfcab251f, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/6654e74727b64370acbfbb6acff70ff2] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp, totalSize=118.9 K 2023-04-28 14:59:37,407 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting edd253c343ee48c7b483a50cd7dfb105, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1682693974134 2023-04-28 14:59:37,407 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting d33570364e1f44098f94b6de805f1685, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682693971961 2023-04-28 14:59:37,407 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting ff4a96be2f234f15ab0158baf42d012a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1682693974790 2023-04-28 14:59:37,407 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 1cfb20a90be9401da2b6b159372f44bd, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682693972020 2023-04-28 14:59:37,408 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 3ac1cf8e19914527af60193d0bf6d440, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1682693976162 2023-04-28 14:59:37,408 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 135518df887a4af1a615d8456c5f95fc, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1682693974134 2023-04-28 14:59:37,408 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 4a9a539bd77140bb83b19bccfcab251f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1682693974790 2023-04-28 14:59:37,408 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 6654e74727b64370acbfbb6acff70ff2, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1682693976162 2023-04-28 14:59:37,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:37,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:37,413 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:37,413 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:37,413 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:37,413 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:37,413 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:37,413 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:37,414 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:37,414 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:37,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:37,414 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,414 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:37,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:37,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:37,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:37,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:37,415 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:37,421 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 2e61d8a0a5c8be60fbc343435192fa13#B#compaction#448 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:59:37,421 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:37,421 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:37,421 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:37,421 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:37,421 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:37,421 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:37,421 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:37,421 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:37,421 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:37,421 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2e61d8a0a5c8be60fbc343435192fa13 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-04-28 14:59:37,422 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=A 2023-04-28 14:59:37,422 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:59:37,422 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=B 2023-04-28 14:59:37,422 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:59:37,422 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=C 2023-04-28 14:59:37,422 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:59:37,423 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2e61d8a0a5c8be60fbc343435192fa13#A#compaction#447 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:59:37,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:37,610 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:37,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:37,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39988 deadline: 1682694037626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:37,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39972 deadline: 1682694037626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,731 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:37,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39988 deadline: 1682694037730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,733 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:37,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39972 deadline: 1682694037731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,839 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/f84ef14288784e078f74dc17b080441a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/f84ef14288784e078f74dc17b080441a 2023-04-28 14:59:37,843 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2e61d8a0a5c8be60fbc343435192fa13/B of 2e61d8a0a5c8be60fbc343435192fa13 into f84ef14288784e078f74dc17b080441a(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:37,843 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:37,843 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13., storeName=2e61d8a0a5c8be60fbc343435192fa13/B, priority=11, startTime=1682693977404; duration=0sec 2023-04-28 14:59:37,843 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:37,843 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 14:59:37,844 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 54742 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 14:59:37,844 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 2e61d8a0a5c8be60fbc343435192fa13/C is initiating minor compaction (all files) 2023-04-28 14:59:37,844 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e61d8a0a5c8be60fbc343435192fa13/C in TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:37,844 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/a2f36f9f09344f638c948bfe5ca77a1f, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/0d4c0e9b0f614b08a35a3e238e6ce331, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/6530f5243efe43e8ab7b410c8d2cfe99, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/3fbc2dc6e9444cf0805caa67dee50bd2, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/a7ff446eefe4498485e64b65be77168e] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp, totalSize=53.5 K 2023-04-28 14:59:37,845 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting a2f36f9f09344f638c948bfe5ca77a1f, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682693971961 2023-04-28 14:59:37,845 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 0d4c0e9b0f614b08a35a3e238e6ce331, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682693972020 2023-04-28 14:59:37,845 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 6530f5243efe43e8ab7b410c8d2cfe99, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1682693974134 2023-04-28 14:59:37,845 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 3fbc2dc6e9444cf0805caa67dee50bd2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1682693974790 2023-04-28 14:59:37,846 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting a7ff446eefe4498485e64b65be77168e, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1682693976162 2023-04-28 14:59:37,853 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 2e61d8a0a5c8be60fbc343435192fa13#C#compaction#450 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:59:37,854 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/d8502a96e7194a66a630f54ea8bde46a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/d8502a96e7194a66a630f54ea8bde46a 2023-04-28 14:59:37,858 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2e61d8a0a5c8be60fbc343435192fa13/A of 2e61d8a0a5c8be60fbc343435192fa13 into d8502a96e7194a66a630f54ea8bde46a(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:37,858 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:37,859 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13., storeName=2e61d8a0a5c8be60fbc343435192fa13/A, priority=11, startTime=1682693977404; duration=0sec 2023-04-28 14:59:37,859 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:37,860 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428cd529552c6d94db8984c5b1b5e2fd162 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428cd529552c6d94db8984c5b1b5e2fd162 2023-04-28 14:59:37,860 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/4b3d78faf65a46f0bd3292ca26b43c31 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/4b3d78faf65a46f0bd3292ca26b43c31 2023-04-28 14:59:37,863 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2e61d8a0a5c8be60fbc343435192fa13/C of 2e61d8a0a5c8be60fbc343435192fa13 into 4b3d78faf65a46f0bd3292ca26b43c31(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:37,863 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:37,863 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13., storeName=2e61d8a0a5c8be60fbc343435192fa13/C, priority=11, startTime=1682693977405; duration=0sec 2023-04-28 14:59:37,864 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:37,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:37,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39988 deadline: 1682694037932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:37,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:37,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39972 deadline: 1682694037934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:38,239 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:38,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39988 deadline: 1682694038238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:38,240 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:38,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39972 deadline: 1682694038238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:38,264 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=143, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/e4e17757ea25428da01bdf480c3bc995 2023-04-28 14:59:38,288 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:38,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694038285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:38,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:38,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:40006 deadline: 1682694038285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:38,289 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:38,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39948 deadline: 1682694038286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:38,672 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/2bb0928c74a540629be04b27b9485bfe 2023-04-28 14:59:38,743 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:38,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39988 deadline: 1682694038741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:38,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:38,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39972 deadline: 1682694038744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:39,080 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/582378578f944e6c846dc37aace5633d 2023-04-28 14:59:39,083 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/e4e17757ea25428da01bdf480c3bc995 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/e4e17757ea25428da01bdf480c3bc995 2023-04-28 14:59:39,085 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/e4e17757ea25428da01bdf480c3bc995, entries=150, sequenceid=143, filesize=25.3 K 2023-04-28 14:59:39,086 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/2bb0928c74a540629be04b27b9485bfe as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/2bb0928c74a540629be04b27b9485bfe 2023-04-28 14:59:39,089 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/2bb0928c74a540629be04b27b9485bfe, entries=150, sequenceid=143, filesize=11.8 K 2023-04-28 14:59:39,089 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/582378578f944e6c846dc37aace5633d as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/582378578f944e6c846dc37aace5633d 2023-04-28 14:59:39,092 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/582378578f944e6c846dc37aace5633d, entries=150, sequenceid=143, filesize=11.8 K 2023-04-28 14:59:39,092 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=60.38 KB/61830 for 2e61d8a0a5c8be60fbc343435192fa13 in 1671ms, sequenceid=143, compaction requested=false 2023-04-28 14:59:39,092 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:39,092 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:39,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:39,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:39,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:39,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:39,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:39,093 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:39,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:39,093 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:39,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:39,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:39,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:39,094 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:39,094 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:39,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:39,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:39,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:39,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:39,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:39,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6efb9ad[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:39,095 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:39,095 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:39,095 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:39,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:39,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:39,095 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:39,095 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:39,096 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:39,095 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:39,096 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:39,096 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:39,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:39,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:39,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:39,096 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:39,096 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:39,096 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:39,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:39,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:39,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:39,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:39,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:39,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:39,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:39,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:39,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:39,098 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:39,099 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:39,099 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:39,099 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:39,099 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:39,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:39,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:39,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:39,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:39,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:39,100 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:39,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:39,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:39,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:39,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:39,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:39,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:39,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:39,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:39,101 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:39,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:39,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:39,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:39,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:39,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:39,102 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:39,102 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:39,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:39,102 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:39,102 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:39,102 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:39,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:39,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:39,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:39,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:39,103 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:39,102 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:39,103 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:39,103 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:39,103 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:39,103 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:39,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:39,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:39,103 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:39,203 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:39,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:39,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:39,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:39,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:39,205 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:39,206 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:39,206 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:39,206 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:39,206 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:39,206 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:39,207 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:39,207 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:39,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:39,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:39,207 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:39,207 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:39,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:39,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:39,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:39,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:39,208 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:39,208 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:39,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:39,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:39,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:39,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:39,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:39,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:39,208 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:39,208 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:39,208 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2e61d8a0a5c8be60fbc343435192fa13 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-28 14:59:39,209 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=A 2023-04-28 14:59:39,209 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:39,209 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=B 2023-04-28 14:59:39,209 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:39,209 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=C 2023-04-28 14:59:39,209 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:39,618 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428dac7446b35354c608b20fa7cc1b7366a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428dac7446b35354c608b20fa7cc1b7366a 2023-04-28 14:59:39,621 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=158, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/debe331f8751464880416185d9c6c48d 2023-04-28 14:59:39,628 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/34fc5b066dda4dbca2f6e47112fcb0d8 2023-04-28 14:59:39,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:39,746 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:39,770 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:39,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39972 deadline: 1682694039769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:39,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:39,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39988 deadline: 1682694039770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:39,872 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:39,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39972 deadline: 1682694039871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:39,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:39,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39988 deadline: 1682694039873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,036 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/2085ae8ba5924ea886d7a21c836eb03e 2023-04-28 14:59:40,039 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/debe331f8751464880416185d9c6c48d as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/debe331f8751464880416185d9c6c48d 2023-04-28 14:59:40,041 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/debe331f8751464880416185d9c6c48d, entries=150, sequenceid=158, filesize=25.3 K 2023-04-28 14:59:40,042 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/34fc5b066dda4dbca2f6e47112fcb0d8 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/34fc5b066dda4dbca2f6e47112fcb0d8 2023-04-28 14:59:40,044 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/34fc5b066dda4dbca2f6e47112fcb0d8, entries=150, sequenceid=158, filesize=11.8 K 2023-04-28 14:59:40,045 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/2085ae8ba5924ea886d7a21c836eb03e as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/2085ae8ba5924ea886d7a21c836eb03e 2023-04-28 14:59:40,047 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/2085ae8ba5924ea886d7a21c836eb03e, entries=150, sequenceid=158, filesize=11.8 K 2023-04-28 14:59:40,048 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 2e61d8a0a5c8be60fbc343435192fa13 in 840ms, sequenceid=158, compaction requested=true 2023-04-28 14:59:40,048 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:40,048 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:40,048 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:40,048 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:40,048 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,048 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:40,048 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:40,049 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,049 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:40,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:40,049 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,049 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:40,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:40,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:40,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:40,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:40,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b329450[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:40,050 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:40,050 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,051 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,051 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:40,051 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,051 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:40,051 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:40,051 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:40,052 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:40,052 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,052 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:40,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:40,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:40,052 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:40,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:40,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:40,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:40,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:40,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:40,054 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:40,054 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:40,054 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:40,054 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:40,054 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:40,056 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,056 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,056 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:40,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:40,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:40,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:40,056 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:40,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:40,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:40,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,057 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:40,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,059 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,059 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,059 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,059 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:40,059 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:40,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:40,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:40,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:40,059 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,059 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,059 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,059 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:40,059 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:40,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:40,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:40,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:40,059 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:40,059 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:40,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:40,076 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2e61d8a0a5c8be60fbc343435192fa13 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-28 14:59:40,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693979767, max timestamp=1682693980075Num uniques 859; 2023-04-28 14:59:40,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:59:40,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693979767, max timestamp=1682693980075Num uniques 859; 2023-04-28 14:59:40,077 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=A 2023-04-28 14:59:40,077 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:59:40,077 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=B 2023-04-28 14:59:40,077 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:59:40,077 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=C 2023-04-28 14:59:40,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:59:40,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693979767, max timestamp=1682693980075Num uniques 859; 2023-04-28 14:59:40,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:59:40,077 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:59:40,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693980091, max timestamp=1682693980099Num uniques 859; 2023-04-28 14:59:40,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:59:40,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693980091, max timestamp=1682693980099Num uniques 859; 2023-04-28 14:59:40,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693980091, max timestamp=1682693980099Num uniques 859; 2023-04-28 14:59:40,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:59:40,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:59:40,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682693980122, max timestamp=1682693980129Num uniques 1068; 2023-04-28 14:59:40,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-28 14:59:40,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7680, totalHeapSize=17712, min timestamp=1682693980122, max timestamp=1682693980129Num uniques 1068; 2023-04-28 14:59:40,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=48900, new segment data size=7680, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-28 14:59:40,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682693980122, max timestamp=1682693980129Num uniques 1068; 2023-04-28 14:59:40,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-28 14:59:40,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682693980144, max timestamp=1682693980157Num uniques 1027; 2023-04-28 14:59:40,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-28 14:59:40,159 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:40,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:40,160 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682693980144, max timestamp=1682693980157Num uniques 1027; 2023-04-28 14:59:40,160 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-28 14:59:40,160 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682693980144, max timestamp=1682693980157Num uniques 1027; 2023-04-28 14:59:40,160 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-28 14:59:40,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:40,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:40,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:40,162 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:40,162 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:40,162 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:40,162 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:40,162 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,162 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,163 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,163 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:40,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:40,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:40,163 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,163 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:40,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:40,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,164 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:40,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:40,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:40,164 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:40,164 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:40,164 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:40,164 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:40,164 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:40,164 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:40,164 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:40,164 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:40,164 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:40,164 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:40,164 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:40,164 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,164 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:40,164 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:40,165 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,165 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:40,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:40,165 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,165 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:40,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:40,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:40,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:40,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:40,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f8e29d9[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:40,166 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:40,167 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,167 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,167 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:40,167 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:40,167 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:40,168 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,168 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:40,168 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,168 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:40,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:40,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:40,168 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:40,168 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:40,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:40,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:40,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:40,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:40,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:40,170 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:40,170 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:40,170 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:40,170 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:40,170 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:40,171 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,171 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:40,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:40,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:40,172 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,171 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:40,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:40,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:40,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:40,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,173 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:40,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,174 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,174 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:40,174 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,174 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:40,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:40,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:40,174 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:40,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:40,174 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:40,175 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:40,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:40,174 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,175 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:40,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:40,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:40,175 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,175 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,175 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682693980199, max timestamp=1682693980211Num uniques 1086; 2023-04-28 14:59:40,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-28 14:59:40,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682693980199, max timestamp=1682693980211Num uniques 1086; 2023-04-28 14:59:40,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-28 14:59:40,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682693980199, max timestamp=1682693980211Num uniques 1086; 2023-04-28 14:59:40,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-28 14:59:40,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682693980223, max timestamp=1682693980243Num uniques 1095; 2023-04-28 14:59:40,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682693980223, max timestamp=1682693980243Num uniques 1095; 2023-04-28 14:59:40,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-28 14:59:40,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-28 14:59:40,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682693980223, max timestamp=1682693980243Num uniques 1095; 2023-04-28 14:59:40,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-28 14:59:40,275 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:40,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:40,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:40,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:40,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:40,277 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:40,277 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:40,277 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:40,278 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:40,278 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,278 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,278 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:40,278 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:40,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:40,279 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,279 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:40,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:40,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,279 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:40,279 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:40,279 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:40,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682693980262, max timestamp=1682693980279Num uniques 1054; 2023-04-28 14:59:40,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-28 14:59:40,281 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:40,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682693980262, max timestamp=1682693980279Num uniques 1054; 2023-04-28 14:59:40,281 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:40,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-28 14:59:40,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682693980262, max timestamp=1682693980279Num uniques 1054; 2023-04-28 14:59:40,281 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:40,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-28 14:59:40,281 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:40,281 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:40,281 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:40,282 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:40,282 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:40,282 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:40,282 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:40,282 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:40,282 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,282 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:40,282 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:40,283 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,283 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:40,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:40,283 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,283 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:40,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:40,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:40,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:40,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:40,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@310f2fd2[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:40,284 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:40,284 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,285 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,285 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:40,285 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:40,285 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:40,285 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,285 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:40,286 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,286 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:40,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:40,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:40,286 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:40,286 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:40,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:40,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:40,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:40,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:40,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:40,287 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:40,287 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:40,287 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:40,287 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:40,287 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:40,288 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:40,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:40,288 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,288 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:40,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:40,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:40,289 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:40,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:40,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,290 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:40,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,291 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:40,291 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:40,291 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:40,291 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:40,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:40,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:40,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:40,291 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,291 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,291 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,291 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,291 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,291 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,291 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:40,291 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:40,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:40,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:40,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1682693980294, max timestamp=1682693980305Num uniques 1063; 2023-04-28 14:59:40,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1682693980294, max timestamp=1682693980305Num uniques 1063; 2023-04-28 14:59:40,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-04-28 14:59:40,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-04-28 14:59:40,308 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1682693980294, max timestamp=1682693980305Num uniques 1063; 2023-04-28 14:59:40,308 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-04-28 14:59:40,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:40,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694040321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:40,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39972 deadline: 1682694040321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1682693980316, max timestamp=1682693980323Num uniques 1122; 2023-04-28 14:59:40,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=14742, suffix heap size=144656,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=322 2023-04-28 14:59:40,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1682693980316, max timestamp=1682693980323Num uniques 1122; 2023-04-28 14:59:40,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=14742, suffix heap size=144656,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=322 2023-04-28 14:59:40,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1682693980316, max timestamp=1682693980323Num uniques 1122; 2023-04-28 14:59:40,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=14742, suffix heap size=144656,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=322 2023-04-28 14:59:40,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:40,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39948 deadline: 1682694040344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:40,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39988 deadline: 1682694040345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:40,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:40006 deadline: 1682694040345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,391 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:40,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:40,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:40,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:40,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:40,394 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:40,394 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:40,394 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:40,394 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:40,395 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,395 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,395 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,395 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:40,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:40,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:40,395 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,395 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:40,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:40,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:40,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:40,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:40,396 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:40,396 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:40,397 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:40,397 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:40,397 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:40,397 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:40,397 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:40,397 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:40,397 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:40,397 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:40,397 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:40,397 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,397 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:40,397 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:40,398 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,398 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:40,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:40,398 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,398 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:40,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:40,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:40,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:40,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:40,399 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:40,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a56ca8f[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:40,399 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,400 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,400 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:40,400 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:40,400 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:40,400 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,400 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:40,400 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:40,400 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,400 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:40,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:40,400 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:40,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:40,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:40,403 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:40,403 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:40,403 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:40,403 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:40,403 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:40,403 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,403 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:40,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,404 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:40,404 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:40,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,405 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:40,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,407 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,407 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:40,407 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:40,407 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:40,407 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:40,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:40,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:40,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:40,407 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,407 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,407 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,407 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:40,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:40,407 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:40,407 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,408 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:40,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:40,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694040425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:40,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39972 deadline: 1682694040426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:40,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39948 deadline: 1682694040447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,450 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:40,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39988 deadline: 1682694040449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,451 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:40,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:40006 deadline: 1682694040449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,485 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042834c84c8b4dbc4c4c9a314431210a86b7 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042834c84c8b4dbc4c4c9a314431210a86b7 2023-04-28 14:59:40,508 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:40,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:40,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:40,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:40,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:40,510 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:40,511 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:40,511 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:40,511 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:40,511 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,511 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,512 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,512 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:40,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:40,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:40,512 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,512 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:40,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:40,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,512 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:40,513 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:40,513 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:40,513 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:40,513 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:40,513 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:40,513 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:40,513 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:40,513 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:40,513 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:40,513 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:40,513 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:40,513 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:40,513 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:40,513 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,513 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:40,513 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:40,514 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,514 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:40,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:40,514 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,514 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:40,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:40,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:40,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:40,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:40,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35395781[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:40,515 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:40,515 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,516 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,516 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:40,516 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:40,516 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:40,516 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,516 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:40,517 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:40,517 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:40,517 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,517 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:40,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:40,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:40,519 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:40,519 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:40,519 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:40,519 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:40,519 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:40,520 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,520 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:40,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:40,520 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:40,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,520 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:40,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:40,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:40,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:40,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,522 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:40,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,523 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,523 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,523 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,523 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,523 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,523 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,523 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:40,523 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:40,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:40,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:40,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:40,523 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:40,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:40,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:40,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:40,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:40,523 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,524 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:40,524 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:40,624 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:40,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:40,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:40,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:40,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:40,627 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:40,627 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:40,627 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:40,627 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:40,627 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,627 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,628 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,630 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:40,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:40,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:40,630 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:40,630 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:40,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694040628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:40,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:40,631 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:40,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39972 deadline: 1682694040629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,631 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:40,631 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:40,631 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:40,631 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:40,631 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:40,632 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:40,632 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:40,632 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:40,632 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:40,632 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:40,632 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:40,632 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:40,632 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:40,632 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,632 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:40,632 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:40,633 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,633 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:40,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:40,633 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,633 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:40,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:40,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:40,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:40,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:40,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bf7701e[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:40,634 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:40,634 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,635 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,635 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:40,635 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:40,635 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:40,635 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,635 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:40,635 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,635 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:40,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:40,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:40,635 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:40,636 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:40,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:40,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:40,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:40,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:40,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:40,637 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:40,637 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:40,637 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:40,638 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:40,638 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:40,638 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,638 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:40,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:40,638 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:40,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:40,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:40,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:40,639 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:40,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,640 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:40,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,641 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,641 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,641 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,641 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:40,641 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:40,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:40,641 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,641 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:40,641 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:40,642 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:40,641 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:40,642 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:40,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:40,642 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:40,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:40,650 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:40,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39948 deadline: 1682694040649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:40,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39988 deadline: 1682694040651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:40,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:40006 deadline: 1682694040653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,742 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:40,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:40,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:40,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:40,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:40,744 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:40,744 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:40,744 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:40,745 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:40,745 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,745 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,745 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,745 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:40,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:40,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:40,746 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,746 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:40,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:40,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:40,747 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:40,747 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:40,747 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:40,747 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:40,747 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:40,747 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:40,747 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:40,747 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:40,747 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:40,747 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:40,747 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:40,747 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:40,747 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:40,747 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,747 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:40,747 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:40,748 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,748 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:40,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:40,748 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,748 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:40,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:40,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:40,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:40,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:40,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@734eb90d[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:40,749 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:40,749 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,750 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,750 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,750 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:40,750 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:40,750 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:40,750 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:40,751 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,751 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:40,751 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:40,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:40,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:40,751 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:40,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:40,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:40,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:40,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:40,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:40,752 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:40,752 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:40,753 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:40,753 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:40,753 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:40,754 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,754 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,754 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:40,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:40,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:40,754 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:40,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:40,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:40,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:40,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,755 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:40,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,757 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,757 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:40,757 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,757 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:40,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:40,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:40,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:40,757 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:40,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:40,757 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:40,757 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:40,757 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,757 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:40,757 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,758 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:40,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:40,758 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,858 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:40,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:40,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:40,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:40,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:40,860 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:40,860 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:40,860 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:40,860 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:40,860 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,860 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,861 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,861 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:40,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:40,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:40,861 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,861 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:40,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:40,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:40,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:40,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:40,862 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:40,862 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:40,862 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:40,862 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:40,862 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:40,862 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:40,862 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:40,862 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:40,862 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:40,862 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:40,863 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:40,863 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,863 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:40,863 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:40,863 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,863 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:40,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:40,864 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,864 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:40,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:40,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:40,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:40,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:40,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f301759[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:40,865 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:40,865 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,865 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,865 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:40,865 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:40,865 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:40,865 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,865 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:40,866 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:40,866 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,866 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:40,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:40,866 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:40,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:40,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:40,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:40,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:40,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:40,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:40,868 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:40,868 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:40,868 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:40,868 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:40,868 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:40,868 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,868 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,868 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:40,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:40,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:40,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,869 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:40,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:40,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:40,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:40,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,870 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:40,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,871 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,871 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,871 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:40,871 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:40,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:40,871 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:40,871 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:40,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:40,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:40,871 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,871 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,871 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,871 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:40,871 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,871 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:40,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:40,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:40,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:40,890 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=180, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/05b578cedafa4f9ea9f2cb403afbb973 2023-04-28 14:59:40,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:40,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39972 deadline: 1682694040932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:40,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694040933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:40,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39948 deadline: 1682694040952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,958 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:40,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39988 deadline: 1682694040955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,959 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:40,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:40006 deadline: 1682694040957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,971 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:40,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:40,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:40,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:40,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:40,974 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:40,974 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:40,974 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:40,974 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:40,974 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,974 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,975 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:40,975 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:40,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:40,975 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,975 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:40,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:40,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:40,976 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:40,976 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:40,976 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:40,976 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:40,976 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:40,976 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:40,977 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:40,977 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:40,977 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:40,977 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:40,977 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:40,977 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:40,977 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:40,977 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,977 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:40,977 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:40,978 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,978 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:40,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:40,978 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,978 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:40,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:40,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:40,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:40,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:40,979 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:40,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@704adc61[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:40,979 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,980 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,980 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:40,980 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:40,980 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:40,980 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,980 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:40,981 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,981 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:40,981 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:40,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:40,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:40,981 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:40,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:40,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:40,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:40,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:40,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:40,983 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:40,983 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:40,983 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:40,983 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:40,983 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:40,984 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,984 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,984 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:40,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:40,984 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:40,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:40,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,985 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:40,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:40,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,986 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,987 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,987 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:40,987 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:40,987 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:40,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:40,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:40,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:40,987 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:40,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:40,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:40,987 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:40,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:40,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:40,987 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,987 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:40,987 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:40,987 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:40,987 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,087 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:41,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:41,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:41,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:41,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:41,090 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:41,090 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:41,090 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:41,090 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:41,090 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,090 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,091 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:41,091 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:41,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:41,091 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,091 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:41,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:41,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:41,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:41,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:41,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:41,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:41,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:41,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:41,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:41,092 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:41,092 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:41,092 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:41,092 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:41,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:41,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:41,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:41,093 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:41,093 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,093 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:41,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:41,093 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,093 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:41,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:41,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:41,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:41,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:41,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70bd25f6[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:41,095 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:41,095 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,095 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,095 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:41,095 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:41,095 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:41,095 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,095 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:41,096 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:41,096 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:41,096 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,096 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:41,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:41,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:41,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:41,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:41,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:41,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:41,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:41,098 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:41,098 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:41,098 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:41,098 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:41,098 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:41,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:41,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:41,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:41,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:41,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:41,099 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:41,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:41,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,100 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:41,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,101 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,101 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,101 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,101 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,101 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,101 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,101 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,101 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:41,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:41,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:41,101 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:41,101 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:41,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:41,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:41,101 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:41,102 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:41,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:41,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:41,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:41,202 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:41,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:41,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:41,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:41,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:41,204 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:41,205 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:41,205 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:41,205 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:41,205 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,205 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,206 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:41,206 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:41,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:41,206 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,206 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:41,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:41,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,206 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:41,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:41,207 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:41,207 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:41,207 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:41,207 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:41,207 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:41,207 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:41,207 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:41,207 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:41,207 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:41,207 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:41,207 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:41,207 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,207 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:41,207 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:41,209 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,209 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:41,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:41,209 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,209 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:41,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:41,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:41,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:41,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:41,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76bd6c3d[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:41,210 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:41,210 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,211 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,211 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:41,211 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:41,211 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,211 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:41,211 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:41,212 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:41,212 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,212 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:41,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:41,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:41,212 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:41,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:41,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:41,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:41,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:41,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:41,214 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:41,214 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:41,214 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:41,214 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:41,214 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:41,214 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,214 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:41,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:41,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,215 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:41,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:41,215 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:41,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:41,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:41,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,216 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:41,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,217 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,217 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:41,217 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:41,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:41,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:41,217 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,218 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:41,218 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:41,218 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:41,217 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:41,217 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:41,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:41,218 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,218 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,218 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:41,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:41,297 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/67658770831c4e13bb4cfd50728fe2db 2023-04-28 14:59:41,318 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:41,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:41,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:41,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:41,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:41,320 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:41,320 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:41,320 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:41,320 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:41,321 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,321 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,321 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,321 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,321 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:41,322 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:41,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:41,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:41,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:41,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,323 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:41,323 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:41,323 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:41,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:41,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:41,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:41,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:41,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:41,323 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:41,323 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:41,323 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:41,323 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:41,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:41,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:41,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:41,324 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:41,324 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,324 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:41,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:41,324 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,324 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:41,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:41,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:41,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:41,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:41,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7938376c[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:41,325 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:41,326 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,326 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,326 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:41,326 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,326 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:41,326 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:41,326 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:41,327 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,327 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:41,327 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:41,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:41,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:41,327 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:41,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:41,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:41,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:41,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:41,329 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:41,329 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:41,329 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:41,329 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:41,329 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:41,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:41,330 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,330 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,330 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:41,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:41,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:41,330 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:41,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:41,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:41,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:41,331 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:41,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,332 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,332 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,332 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,332 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:41,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:41,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:41,332 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:41,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:41,332 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:41,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:41,332 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:41,333 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:41,333 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,333 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:41,333 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:41,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:41,333 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,433 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:41,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:41,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:41,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:41,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:41,435 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:41,436 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:41,436 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:41,436 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:41,436 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,436 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,437 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:41,437 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:41,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:41,437 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,437 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:41,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:41,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:41,438 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:41,438 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:41,438 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:41,438 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:41,438 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:41,438 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:41,438 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:41,438 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:41,438 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:41,438 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:41,438 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:41,438 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:41,438 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:41,438 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,438 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:41,438 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:41,439 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,439 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:41,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:41,439 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,439 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:41,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:41,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:41,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:41,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39972 deadline: 1682694041438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:41,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:41,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fb745fe[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:41,440 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:41,441 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,441 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,441 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:41,441 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:41,441 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:41,441 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,441 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:41,442 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:41,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694041440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,442 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,442 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:41,442 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:41,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:41,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:41,442 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:41,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:41,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:41,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:41,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:41,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:41,444 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:41,444 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:41,444 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:41,444 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:41,444 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:41,445 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,445 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,445 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:41,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:41,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:41,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:41,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:41,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:41,446 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:41,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,446 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:41,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,448 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,448 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,448 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,448 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,448 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,448 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,448 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:41,448 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:41,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:41,448 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:41,448 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:41,450 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:41,450 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:41,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:41,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:41,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:41,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:41,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:41,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39948 deadline: 1682694041459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,464 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:41,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39988 deadline: 1682694041461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:41,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:40006 deadline: 1682694041462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,548 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:41,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:41,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:41,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:41,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:41,552 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:41,553 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:41,553 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:41,553 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:41,553 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,553 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,554 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,554 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:41,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:41,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:41,554 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,554 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:41,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:41,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,554 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:41,555 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:41,555 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:41,555 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:41,555 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:41,555 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:41,555 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:41,555 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:41,555 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:41,555 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:41,555 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:41,555 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:41,555 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:41,555 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:41,555 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,555 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:41,555 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:41,556 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,556 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:41,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:41,556 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,556 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:41,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:41,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:41,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:41,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:41,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@463c496b[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:41,557 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:41,557 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,558 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,559 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:41,559 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:41,559 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:41,559 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,559 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:41,559 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:41,559 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,559 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:41,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:41,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:41,559 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:41,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:41,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:41,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:41,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:41,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:41,561 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:41,561 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:41,561 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:41,561 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:41,561 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:41,562 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,562 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,562 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:41,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:41,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:41,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:41,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:41,562 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:41,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:41,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,563 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:41,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,564 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,564 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,564 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,564 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,564 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,564 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:41,564 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:41,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:41,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:41,564 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,564 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:41,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:41,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:41,565 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:41,565 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:41,565 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:41,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:41,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:41,665 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:41,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:41,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:41,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:41,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:41,667 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:41,667 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:41,667 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:41,667 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:41,668 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,668 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,668 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:41,668 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:41,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:41,669 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,669 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:41,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:41,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,669 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:41,669 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:41,669 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:41,670 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:41,670 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:41,670 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:41,670 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:41,670 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:41,670 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:41,670 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:41,670 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:41,670 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:41,670 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:41,670 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:41,670 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,670 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:41,670 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:41,671 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,671 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:41,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:41,671 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,671 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:41,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:41,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:41,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:41,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:41,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b7d1915[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:41,672 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:41,672 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,672 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,673 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:41,673 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,673 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:41,673 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:41,673 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:41,673 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,673 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:41,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:41,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:41,673 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:41,674 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:41,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:41,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:41,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:41,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:41,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:41,675 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:41,675 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:41,675 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:41,675 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:41,675 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:41,676 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:41,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:41,676 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,676 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:41,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:41,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:41,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:41,677 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:41,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,678 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:41,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:41,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,679 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,679 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,679 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:41,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:41,679 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:41,679 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,679 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:41,679 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:41,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:41,679 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:41,679 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:41,679 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:41,679 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:41,679 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:41,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:41,705 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/8a61fb65bd0846dd81d2662415e0f4ec 2023-04-28 14:59:41,708 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/05b578cedafa4f9ea9f2cb403afbb973 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/05b578cedafa4f9ea9f2cb403afbb973 2023-04-28 14:59:41,710 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/05b578cedafa4f9ea9f2cb403afbb973, entries=200, sequenceid=180, filesize=32.1 K 2023-04-28 14:59:41,711 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/67658770831c4e13bb4cfd50728fe2db as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/67658770831c4e13bb4cfd50728fe2db 2023-04-28 14:59:41,713 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/67658770831c4e13bb4cfd50728fe2db, entries=150, sequenceid=180, filesize=11.8 K 2023-04-28 14:59:41,714 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/8a61fb65bd0846dd81d2662415e0f4ec as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/8a61fb65bd0846dd81d2662415e0f4ec 2023-04-28 14:59:41,716 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/8a61fb65bd0846dd81d2662415e0f4ec, entries=150, sequenceid=180, filesize=11.8 K 2023-04-28 14:59:41,717 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=154.31 KB/158010 for 2e61d8a0a5c8be60fbc343435192fa13 in 1641ms, sequenceid=180, compaction requested=true 2023-04-28 14:59:41,717 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:41,717 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:41,717 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:59:41,717 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:59:41,717 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:41,717 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:41,718 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:59:41,718 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): 2e61d8a0a5c8be60fbc343435192fa13/A is initiating minor compaction (all files) 2023-04-28 14:59:41,718 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48166 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:59:41,718 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e61d8a0a5c8be60fbc343435192fa13/A in TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:41,718 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 2e61d8a0a5c8be60fbc343435192fa13/B is initiating minor compaction (all files) 2023-04-28 14:59:41,718 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e61d8a0a5c8be60fbc343435192fa13/B in TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:41,718 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/d8502a96e7194a66a630f54ea8bde46a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/e4e17757ea25428da01bdf480c3bc995, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/debe331f8751464880416185d9c6c48d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/05b578cedafa4f9ea9f2cb403afbb973] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp, totalSize=108.0 K 2023-04-28 14:59:41,718 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/f84ef14288784e078f74dc17b080441a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/2bb0928c74a540629be04b27b9485bfe, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/34fc5b066dda4dbca2f6e47112fcb0d8, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/67658770831c4e13bb4cfd50728fe2db] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp, totalSize=47.0 K 2023-04-28 14:59:41,719 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting f84ef14288784e078f74dc17b080441a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1682693974796 2023-04-28 14:59:41,719 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting d8502a96e7194a66a630f54ea8bde46a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1682693974796 2023-04-28 14:59:41,719 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 2bb0928c74a540629be04b27b9485bfe, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1682693976985 2023-04-28 14:59:41,719 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting e4e17757ea25428da01bdf480c3bc995, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1682693976985 2023-04-28 14:59:41,719 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting debe331f8751464880416185d9c6c48d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1682693977617 2023-04-28 14:59:41,719 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 34fc5b066dda4dbca2f6e47112fcb0d8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1682693977617 2023-04-28 14:59:41,720 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 05b578cedafa4f9ea9f2cb403afbb973, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1682693979767 2023-04-28 14:59:41,720 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 67658770831c4e13bb4cfd50728fe2db, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1682693979767 2023-04-28 14:59:41,727 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 2e61d8a0a5c8be60fbc343435192fa13#B#compaction#460 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:59:41,729 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2e61d8a0a5c8be60fbc343435192fa13#A#compaction#459 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:59:41,780 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:41,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:41,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:41,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:41,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:41,782 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:41,782 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:41,782 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:41,782 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:41,782 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,782 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,783 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:41,783 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:41,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:41,783 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:41,783 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:41,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:41,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:41,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:41,784 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:41,784 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:41,784 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:41,784 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:41,784 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:41,784 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:41,784 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:41,784 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:41,784 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:41,784 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:41,784 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2e61d8a0a5c8be60fbc343435192fa13 3/3 column families, dataSize=154.31 KB heapSize=429.47 KB 2023-04-28 14:59:41,785 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=A 2023-04-28 14:59:41,785 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:59:41,785 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=B 2023-04-28 14:59:41,785 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:59:41,785 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=C 2023-04-28 14:59:41,785 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:59:42,134 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/ef20816bc979480086a069ace82c9199 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/ef20816bc979480086a069ace82c9199 2023-04-28 14:59:42,137 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2e61d8a0a5c8be60fbc343435192fa13/B of 2e61d8a0a5c8be60fbc343435192fa13 into ef20816bc979480086a069ace82c9199(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:42,137 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:42,137 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13., storeName=2e61d8a0a5c8be60fbc343435192fa13/B, priority=12, startTime=1682693981717; duration=0sec 2023-04-28 14:59:42,137 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:42,137 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:59:42,138 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48166 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:59:42,138 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 2e61d8a0a5c8be60fbc343435192fa13/C is initiating minor compaction (all files) 2023-04-28 14:59:42,139 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e61d8a0a5c8be60fbc343435192fa13/C in TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:42,139 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/4b3d78faf65a46f0bd3292ca26b43c31, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/582378578f944e6c846dc37aace5633d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/2085ae8ba5924ea886d7a21c836eb03e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/8a61fb65bd0846dd81d2662415e0f4ec] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp, totalSize=47.0 K 2023-04-28 14:59:42,139 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 4b3d78faf65a46f0bd3292ca26b43c31, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1682693974796 2023-04-28 14:59:42,139 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 582378578f944e6c846dc37aace5633d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1682693976985 2023-04-28 14:59:42,140 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 2085ae8ba5924ea886d7a21c836eb03e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1682693977617 2023-04-28 14:59:42,140 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 8a61fb65bd0846dd81d2662415e0f4ec, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1682693979767 2023-04-28 14:59:42,144 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 2e61d8a0a5c8be60fbc343435192fa13#C#compaction#462 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:59:42,195 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428a7ae4976a7b843558b8d974c7a23149b to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a7ae4976a7b843558b8d974c7a23149b 2023-04-28 14:59:42,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:42,445 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:42,451 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:42,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694042449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:42,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:42,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39972 deadline: 1682694042451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:42,468 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:42,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39948 deadline: 1682694042466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:42,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:42,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:40006 deadline: 1682694042468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:42,473 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:42,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39988 deadline: 1682694042473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:42,542 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/31fde17a97e342d9808ffb07799c6dd1 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/31fde17a97e342d9808ffb07799c6dd1 2023-04-28 14:59:42,545 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2e61d8a0a5c8be60fbc343435192fa13/A of 2e61d8a0a5c8be60fbc343435192fa13 into 31fde17a97e342d9808ffb07799c6dd1(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:42,545 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:42,545 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13., storeName=2e61d8a0a5c8be60fbc343435192fa13/A, priority=12, startTime=1682693981717; duration=0sec 2023-04-28 14:59:42,545 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:42,552 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/5392f5a04aab413db26535f541b26e2d as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/5392f5a04aab413db26535f541b26e2d 2023-04-28 14:59:42,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:42,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694042552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:42,555 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:42,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39972 deadline: 1682694042554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:42,555 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2e61d8a0a5c8be60fbc343435192fa13/C of 2e61d8a0a5c8be60fbc343435192fa13 into 5392f5a04aab413db26535f541b26e2d(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:42,556 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:42,556 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13., storeName=2e61d8a0a5c8be60fbc343435192fa13/C, priority=12, startTime=1682693981717; duration=0sec 2023-04-28 14:59:42,556 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:42,599 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=337, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/4502c11ca9ca454aae4feeb6c47eefdf 2023-04-28 14:59:42,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:42,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694042757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:42,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:42,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39972 deadline: 1682694042757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,007 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/1573848cf35346b582c90a15c494bbbd 2023-04-28 14:59:43,014 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/9f025ae910e349f99b4b92dc145eb726 2023-04-28 14:59:43,017 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/4502c11ca9ca454aae4feeb6c47eefdf as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/4502c11ca9ca454aae4feeb6c47eefdf 2023-04-28 14:59:43,020 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/4502c11ca9ca454aae4feeb6c47eefdf, entries=150, sequenceid=337, filesize=25.4 K 2023-04-28 14:59:43,020 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/1573848cf35346b582c90a15c494bbbd as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/1573848cf35346b582c90a15c494bbbd 2023-04-28 14:59:43,023 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/1573848cf35346b582c90a15c494bbbd, entries=150, sequenceid=337, filesize=11.9 K 2023-04-28 14:59:43,023 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/9f025ae910e349f99b4b92dc145eb726 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/9f025ae910e349f99b4b92dc145eb726 2023-04-28 14:59:43,026 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/9f025ae910e349f99b4b92dc145eb726, entries=150, sequenceid=337, filesize=11.9 K 2023-04-28 14:59:43,028 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~429.42 KB/439728, currentSize=33.54 KB/34350 for 2e61d8a0a5c8be60fbc343435192fa13 in 1244ms, sequenceid=337, compaction requested=false 2023-04-28 14:59:43,028 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:43,028 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:43,028 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:43,028 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:43,028 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,028 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:43,028 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:43,029 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,029 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:43,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:43,029 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,029 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:43,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:43,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:43,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:43,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:43,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6346d641[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:43,031 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:43,031 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,031 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,031 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:43,031 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:43,031 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:43,031 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,031 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:43,032 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:43,032 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:43,032 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:43,032 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:43,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:43,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:43,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:43,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:43,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:43,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:43,035 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:43,035 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:43,035 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:43,035 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:43,035 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:43,036 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,036 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:43,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:43,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:43,036 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:43,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:43,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:43,036 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:43,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,037 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:43,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,039 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:43,039 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:43,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:43,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:43,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:43,039 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:43,039 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,039 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,039 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,039 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,039 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,039 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:43,039 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:43,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:43,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:43,039 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,039 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:43,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:43,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:43,062 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2e61d8a0a5c8be60fbc343435192fa13 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 14:59:43,063 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=A 2023-04-28 14:59:43,063 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:43,063 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=B 2023-04-28 14:59:43,063 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:43,063 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=C 2023-04-28 14:59:43,063 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:43,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693983094, max timestamp=1682693983098Num uniques 859; 2023-04-28 14:59:43,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:59:43,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693983094, max timestamp=1682693983098Num uniques 859; 2023-04-28 14:59:43,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693983094, max timestamp=1682693983098Num uniques 859; 2023-04-28 14:59:43,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:59:43,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:59:43,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:43,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39972 deadline: 1682694043122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:43,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694043123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,139 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:43,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:43,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:43,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:43,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:43,142 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:43,142 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:43,142 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:43,142 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:43,143 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,143 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,143 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,143 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:43,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:43,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:43,144 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,144 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:43,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:43,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,144 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:43,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:43,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:43,144 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:43,144 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:43,144 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:43,145 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:43,145 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:43,145 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:43,145 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:43,145 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:43,145 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:43,145 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:43,145 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:43,145 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,145 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:43,145 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:43,146 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,146 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:43,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:43,146 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,146 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:43,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:43,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:43,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:43,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:43,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63bf762[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:43,147 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:43,147 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,147 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,147 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:43,148 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:43,148 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,148 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:43,148 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:43,149 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,149 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:43,149 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:43,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:43,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:43,149 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:43,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:43,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:43,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:43,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:43,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:43,151 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:43,151 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:43,151 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:43,151 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:43,151 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:43,151 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,152 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:43,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:43,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,152 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:43,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:43,152 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:43,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:43,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:43,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,153 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:43,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,154 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,154 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,154 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:43,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:43,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:43,154 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:43,154 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:43,154 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:43,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:43,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:43,154 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:43,154 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,154 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:43,154 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,154 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,154 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:43,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:43,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:43,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39972 deadline: 1682694043227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:43,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694043227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,254 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:43,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:43,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:43,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:43,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:43,257 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:43,257 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:43,257 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:43,257 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:43,258 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,258 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,258 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,258 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:43,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:43,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:43,258 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,258 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:43,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:43,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,259 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:43,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:43,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:43,259 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:43,259 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:43,260 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:43,260 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:43,260 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:43,260 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:43,260 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:43,260 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:43,260 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:43,260 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:43,260 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:43,260 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,260 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:43,260 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:43,261 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,261 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:43,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:43,261 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:43,261 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:43,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:43,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:43,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:43,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22e881df[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:43,262 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:43,262 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,262 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,263 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,263 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:43,263 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:43,263 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:43,263 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:43,263 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,263 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:43,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:43,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:43,263 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:43,263 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:43,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:43,265 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:43,265 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:43,265 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:43,265 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:43,265 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:43,266 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,266 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,266 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:43,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:43,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:43,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:43,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:43,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:43,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,267 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:43,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,268 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:43,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,269 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,269 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:43,269 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,269 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:43,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:43,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:43,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:43,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:43,269 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:43,269 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,270 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,270 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,270 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:43,270 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,270 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:43,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:43,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:43,270 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,370 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:43,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:43,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:43,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:43,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:43,373 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:43,373 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:43,373 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:43,373 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:43,373 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,373 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,374 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:43,374 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:43,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:43,374 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,374 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:43,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,375 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:43,375 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:43,375 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:43,375 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:43,375 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:43,375 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:43,375 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:43,375 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:43,375 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:43,375 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:43,375 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:43,375 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:43,375 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:43,375 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:43,375 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,375 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:43,375 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:43,376 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:43,376 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:43,376 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,376 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:43,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:43,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@116e8126[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:43,377 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:43,377 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,378 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,378 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:43,378 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:43,378 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:43,378 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,378 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:43,379 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:43,379 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:43,379 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:43,379 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:43,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:43,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:43,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:43,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:43,381 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:43,381 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:43,381 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:43,381 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:43,381 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:43,384 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,384 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,384 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:43,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:43,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:43,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:43,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:43,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:43,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,384 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:43,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,385 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:43,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,387 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:43,387 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:43,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:43,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:43,387 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:43,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:43,388 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:43,387 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:43,387 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,388 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:43,388 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,388 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:43,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:43,388 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,388 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,388 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:43,432 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:43,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694043429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,432 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:43,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39972 deadline: 1682694043430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,475 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304281411ae3fcd0d4d04927784b0ec99c09b to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304281411ae3fcd0d4d04927784b0ec99c09b 2023-04-28 14:59:43,488 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:43,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:43,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:43,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:43,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:43,490 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:43,490 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:43,491 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:43,491 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:43,491 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,491 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,492 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:43,492 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:43,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:43,492 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,492 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:43,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:43,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,493 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:43,493 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:43,493 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:43,493 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:43,493 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:43,493 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:43,493 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:43,493 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:43,493 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:43,493 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:43,493 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:43,494 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:43,494 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:43,494 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:43,494 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,494 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:43,494 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:43,494 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,494 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:43,495 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,495 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:43,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:43,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44cfa994[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:43,496 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:43,496 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,499 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,499 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:43,499 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:43,499 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,499 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:43,499 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:43,500 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:43,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:43,500 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:43,500 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:43,500 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:43,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:43,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:43,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:43,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:43,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:43,502 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:43,502 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:43,502 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:43,502 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:43,502 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:43,504 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:43,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:43,505 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,505 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:43,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:43,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,505 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:43,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:43,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:43,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,506 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:43,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,508 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:43,509 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:43,509 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:43,509 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:43,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:43,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:43,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:43,509 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,509 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,509 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,509 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:43,509 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:43,509 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,509 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:43,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:43,509 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,609 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:43,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:43,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:43,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:43,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:43,612 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:43,612 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:43,612 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:43,612 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:43,612 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,612 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,613 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:43,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:43,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:43,613 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,613 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,613 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:43,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:43,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:43,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:43,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:43,614 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:43,614 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:43,614 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:43,614 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:43,614 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:43,614 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:43,614 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:43,614 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:43,614 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:43,614 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:43,615 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:43,615 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,615 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:43,615 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:43,615 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,615 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:43,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:43,616 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,616 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:43,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:43,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:43,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:43,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:43,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d16c688[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:43,617 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:43,617 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,618 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,618 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,618 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:43,618 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:43,618 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:43,618 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:43,619 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:43,619 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:43,619 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:43,619 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:43,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:43,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:43,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:43,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:43,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:43,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:43,621 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:43,621 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:43,621 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:43,621 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:43,621 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:43,622 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,622 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:43,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,622 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:43,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:43,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:43,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:43,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,623 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:43,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:43,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,624 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:43,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,626 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,626 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,626 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,626 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:43,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:43,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:43,626 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:43,626 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:43,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:43,626 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,626 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:43,626 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,626 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:43,626 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,626 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:43,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:43,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:43,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:43,726 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:43,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:43,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:43,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:43,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:43,729 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:43,729 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:43,729 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:43,729 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:43,729 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,729 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,730 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,730 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:43,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:43,730 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,730 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:43,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,731 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:43,731 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:43,731 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:43,731 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:43,731 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:43,731 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:43,731 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:43,731 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:43,731 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:43,731 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:43,731 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:43,732 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:43,732 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:43,732 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:43,732 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,732 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:43,732 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:43,732 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,732 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:43,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:43,733 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,733 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:43,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:43,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:43,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:43,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:43,734 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:43,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b965a8c[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:43,734 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,734 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:43,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39972 deadline: 1682694043733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,735 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,735 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:43,735 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:43,735 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:43,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:43,735 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694043734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,735 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:43,735 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,736 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:43,736 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:43,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:43,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:43,736 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:43,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:43,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:43,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:43,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:43,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:43,737 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:43,738 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:43,738 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:43,738 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:43,738 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:43,739 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,739 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:43,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:43,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,739 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:43,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:43,739 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:43,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:43,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:43,740 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:43,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,741 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:43,741 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:43,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:43,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:43,741 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,742 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,742 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,742 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:43,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:43,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:43,741 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:43,742 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:43,742 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:43,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:43,742 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,743 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,743 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:43,843 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:43,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:43,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:43,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:43,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:43,846 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:43,846 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:43,846 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:43,846 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:43,846 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,846 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,847 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:43,847 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:43,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:43,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,847 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,847 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:43,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:43,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,848 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:43,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:43,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:43,848 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:43,848 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:43,848 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:43,848 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:43,848 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:43,848 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:43,848 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:43,848 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:43,848 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:43,848 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:43,848 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:43,849 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,849 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:43,849 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:43,849 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,849 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:43,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:43,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:43,850 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,850 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:43,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:43,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:43,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:43,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e948369[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:43,851 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:43,851 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,851 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,851 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:43,851 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:43,852 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,852 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:43,852 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:43,852 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,852 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:43,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:43,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:43,852 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:43,852 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:43,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:43,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:43,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:43,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:43,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:43,854 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:43,854 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:43,854 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:43,854 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:43,854 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:43,855 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:43,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:43,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:43,855 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,863 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,863 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:43,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:43,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:43,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:43,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:43,864 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,866 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,866 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:43,866 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:43,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:43,866 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,866 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,866 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:43,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:43,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:43,866 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,866 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,866 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,866 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:43,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:43,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:43,867 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:43,867 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:43,880 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=350, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/b57a90b36764476b8fab1c53ce6e4f1e 2023-04-28 14:59:43,967 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:43,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:43,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:43,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:43,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:43,971 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:43,971 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:43,971 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:43,971 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:43,972 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,972 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,973 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,973 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:43,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:43,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:43,973 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,973 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:43,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:43,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,974 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:43,974 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:43,974 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:43,974 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:43,974 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:43,974 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:43,975 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:43,975 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:43,975 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:43,975 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:43,975 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:43,975 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:43,975 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:43,975 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:43,975 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,975 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:43,975 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:43,976 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,976 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:43,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:43,976 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,976 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:43,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:43,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:43,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:43,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:43,977 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:43,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@272e0a26[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:43,977 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,978 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,978 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,978 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:43,978 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:43,978 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:43,978 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:43,979 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,979 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:43,979 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:43,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:43,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:43,979 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:43,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:43,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:43,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:43,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:43,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:43,981 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:43,981 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:43,981 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:43,981 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:43,981 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:43,982 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,982 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:43,982 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:43,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:43,982 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:43,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:43,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:43,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:43,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:43,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,984 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:43,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:43,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,986 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:43,986 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,986 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:43,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:43,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:43,986 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:43,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:43,986 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:43,986 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:43,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:43,986 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:43,986 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:43,986 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:43,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:43,986 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:43,986 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:43,986 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,086 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:44,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:44,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:44,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:44,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:44,089 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:44,089 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:44,089 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:44,089 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:44,089 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,089 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,090 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:44,090 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:44,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:44,090 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,090 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:44,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:44,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,091 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:44,091 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:44,091 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:44,091 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:44,091 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:44,091 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:44,091 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:44,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:44,092 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:44,092 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:44,092 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:44,092 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:44,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:44,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:44,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:44,092 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:44,093 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,093 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:44,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:44,093 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,093 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:44,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:44,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:44,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:44,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:44,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ca7aacd[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:44,094 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:44,094 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,095 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,095 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:44,095 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:44,095 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:44,095 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,095 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:44,095 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,095 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:44,095 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:44,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:44,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:44,096 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:44,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:44,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:44,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:44,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:44,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:44,098 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:44,098 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:44,098 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:44,098 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:44,098 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:44,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:44,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:44,099 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:44,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:44,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:44,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,100 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:44,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:44,100 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:44,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,102 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,102 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,102 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:44,102 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,102 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:44,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:44,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:44,102 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:44,102 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:44,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:44,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:44,102 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:44,102 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:44,102 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,102 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:44,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:44,102 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,202 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:44,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:44,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:44,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:44,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:44,205 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:44,205 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:44,205 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:44,205 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:44,205 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,205 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,206 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:44,206 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:44,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:44,206 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,206 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:44,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:44,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,207 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:44,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:44,207 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:44,207 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:44,207 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:44,207 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:44,207 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:44,207 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:44,207 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:44,208 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:44,208 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:44,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:44,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:44,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:44,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:44,209 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,209 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:44,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:44,209 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,209 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:44,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:44,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:44,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:44,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:44,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@452cc3ef[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:44,210 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:44,210 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,211 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,211 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:44,211 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,211 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:44,211 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:44,211 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:44,212 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,212 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:44,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:44,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:44,212 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:44,212 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:44,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:44,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:44,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:44,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:44,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:44,214 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:44,214 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:44,214 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:44,214 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:44,214 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:44,215 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,215 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:44,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:44,215 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:44,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:44,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:44,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:44,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,216 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:44,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,216 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:44,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,218 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,218 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,218 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,218 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,218 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:44,218 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,218 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,218 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,218 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:44,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:44,218 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:44,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:44,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:44,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:44,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:44,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:44,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:44,219 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:44,219 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:44,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:44,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39972 deadline: 1682694044237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:44,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694044240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,287 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=350 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/39d4e5de7c724503b34bc1ee73cb4105 2023-04-28 14:59:44,319 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:44,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:44,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:44,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:44,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:44,321 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:44,321 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:44,321 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:44,321 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:44,322 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,322 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,322 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,322 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:44,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:44,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:44,322 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,322 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:44,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:44,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,323 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:44,323 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:44,323 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:44,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:44,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:44,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:44,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:44,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:44,323 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:44,323 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:44,324 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:44,324 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:44,324 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:44,324 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:44,324 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,324 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:44,324 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:44,324 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,324 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:44,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:44,325 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,325 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:44,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:44,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:44,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:44,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:44,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f198fe8[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:44,326 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:44,326 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,326 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,326 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:44,326 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:44,326 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,326 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:44,326 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:44,327 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,327 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:44,327 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:44,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:44,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:44,327 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:44,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:44,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:44,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:44,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:44,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:44,329 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:44,329 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:44,329 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:44,329 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:44,329 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:44,329 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,329 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:44,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:44,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,329 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:44,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:44,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:44,330 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:44,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:44,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,330 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:44,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,332 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,332 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,332 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,332 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:44,332 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:44,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:44,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:44,332 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:44,332 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,332 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,332 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,332 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:44,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:44,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:44,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:44,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:44,332 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:44,332 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:44,432 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:44,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:44,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:44,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:44,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:44,435 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:44,435 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:44,435 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:44,435 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:44,435 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,435 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,436 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,436 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:44,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:44,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:44,436 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,436 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:44,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:44,437 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:44,437 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:44,437 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:44,437 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:44,437 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:44,437 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:44,437 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:44,437 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:44,437 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:44,438 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:44,438 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:44,438 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:44,438 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:44,438 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,438 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:44,438 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:44,439 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,439 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:44,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:44,439 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,439 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:44,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:44,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:44,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:44,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:44,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2831d517[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:44,440 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:44,440 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,441 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,441 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:44,441 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:44,441 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,441 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:44,441 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:44,442 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,442 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:44,442 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:44,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:44,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:44,442 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:44,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:44,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:44,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:44,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:44,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:44,444 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:44,444 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:44,444 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:44,444 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:44,444 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:44,445 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,445 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,445 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:44,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:44,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:44,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:44,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:44,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:44,445 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:44,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,446 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:44,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,447 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,447 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,447 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,447 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:44,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:44,447 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:44,447 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,448 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:44,448 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:44,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:44,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:44,448 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,448 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:44,448 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:44,448 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:44,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:44,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:44,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:44,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39948 deadline: 1682694044473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,476 DEBUG [Thread-3272] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4131 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13., hostname=jenkins-hbase17.apache.org,37181,1682693812482, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 14:59:44,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:44,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:40006 deadline: 1682694044483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,485 DEBUG [Thread-3276] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13., hostname=jenkins-hbase17.apache.org,37181,1682693812482, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 14:59:44,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:44,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39988 deadline: 1682694044490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,492 DEBUG [Thread-3278] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4146 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13., hostname=jenkins-hbase17.apache.org,37181,1682693812482, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 14:59:44,548 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:44,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:44,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:44,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:44,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:44,551 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:44,551 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:44,551 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:44,551 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:44,551 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,551 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,552 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:44,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:44,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:44,552 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,553 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,553 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:44,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:44,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:44,553 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:44,553 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:44,553 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:44,553 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:44,553 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:44,554 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:44,554 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:44,554 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:44,554 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:44,554 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:44,554 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:44,554 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:44,554 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:44,554 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,554 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:44,554 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:44,555 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,555 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:44,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:44,555 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,555 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:44,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:44,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:44,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:44,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:44,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1971f31e[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:44,556 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:44,556 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,557 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,557 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:44,557 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:44,557 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,557 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:44,557 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:44,558 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:44,558 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,558 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:44,558 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:44,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:44,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:44,560 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:44,560 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:44,560 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:44,560 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:44,560 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:44,561 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,561 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:44,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,561 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:44,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:44,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:44,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:44,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:44,562 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:44,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,563 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:44,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,564 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,564 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,564 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,564 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,564 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,564 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:44,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:44,564 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:44,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:44,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:44,564 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:44,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:44,564 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:44,565 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:44,565 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:44,564 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:44,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:44,665 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:44,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:44,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:44,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:44,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:44,667 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:44,667 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:44,667 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:44,667 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:44,668 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,668 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,668 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:44,668 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:44,669 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,669 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:44,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:44,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,669 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:44,669 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:44,669 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:44,669 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:44,669 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:44,669 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:44,670 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:44,670 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:44,670 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:44,670 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:44,670 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:44,670 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:44,670 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:44,670 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:44,670 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,670 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:44,670 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:44,671 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,671 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:44,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:44,672 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,672 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:44,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:44,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:44,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:44,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:44,673 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:44,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a617864[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:44,673 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,673 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,673 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:44,673 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,674 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:44,673 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:44,674 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:44,674 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:44,674 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,674 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:44,674 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:44,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:44,676 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:44,676 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:44,676 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:44,676 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:44,676 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:44,677 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,677 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:44,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:44,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,677 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:44,677 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:44,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:44,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:44,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:44,678 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:44,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:44,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,679 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,679 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:44,680 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:44,679 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:44,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:44,680 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:44,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:44,680 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:44,680 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:44,680 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:44,680 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,680 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,680 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:44,680 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:44,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:44,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:44,695 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=350 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/19d3903afe35457097f69e4c326db980 2023-04-28 14:59:44,698 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/b57a90b36764476b8fab1c53ce6e4f1e as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/b57a90b36764476b8fab1c53ce6e4f1e 2023-04-28 14:59:44,701 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/b57a90b36764476b8fab1c53ce6e4f1e, entries=100, sequenceid=350, filesize=18.6 K 2023-04-28 14:59:44,701 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/39d4e5de7c724503b34bc1ee73cb4105 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/39d4e5de7c724503b34bc1ee73cb4105 2023-04-28 14:59:44,704 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/39d4e5de7c724503b34bc1ee73cb4105, entries=100, sequenceid=350, filesize=9.5 K 2023-04-28 14:59:44,704 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/19d3903afe35457097f69e4c326db980 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/19d3903afe35457097f69e4c326db980 2023-04-28 14:59:44,707 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/19d3903afe35457097f69e4c326db980, entries=100, sequenceid=350, filesize=9.5 K 2023-04-28 14:59:44,707 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2e61d8a0a5c8be60fbc343435192fa13 in 1645ms, sequenceid=350, compaction requested=true 2023-04-28 14:59:44,707 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:44,707 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:44,707 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:44,707 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:59:44,707 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:59:44,707 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:44,708 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34272 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:59:44,708 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 2e61d8a0a5c8be60fbc343435192fa13/B is initiating minor compaction (all files) 2023-04-28 14:59:44,708 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 71248 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:59:44,708 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e61d8a0a5c8be60fbc343435192fa13/B in TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:44,708 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): 2e61d8a0a5c8be60fbc343435192fa13/A is initiating minor compaction (all files) 2023-04-28 14:59:44,708 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/ef20816bc979480086a069ace82c9199, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/1573848cf35346b582c90a15c494bbbd, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/39d4e5de7c724503b34bc1ee73cb4105] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp, totalSize=33.5 K 2023-04-28 14:59:44,708 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e61d8a0a5c8be60fbc343435192fa13/A in TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:44,708 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/31fde17a97e342d9808ffb07799c6dd1, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/4502c11ca9ca454aae4feeb6c47eefdf, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/b57a90b36764476b8fab1c53ce6e4f1e] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp, totalSize=69.6 K 2023-04-28 14:59:44,708 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting ef20816bc979480086a069ace82c9199, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1682693979767 2023-04-28 14:59:44,709 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 31fde17a97e342d9808ffb07799c6dd1, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1682693979767 2023-04-28 14:59:44,709 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 1573848cf35346b582c90a15c494bbbd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1682693980341 2023-04-28 14:59:44,709 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 39d4e5de7c724503b34bc1ee73cb4105, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1682693983061 2023-04-28 14:59:44,709 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 4502c11ca9ca454aae4feeb6c47eefdf, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1682693980341 2023-04-28 14:59:44,709 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting b57a90b36764476b8fab1c53ce6e4f1e, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1682693983061 2023-04-28 14:59:44,713 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 2e61d8a0a5c8be60fbc343435192fa13#B#compaction#468 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:59:44,715 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2e61d8a0a5c8be60fbc343435192fa13#A#compaction#469 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:59:44,780 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:44,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:44,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:44,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:44,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:44,782 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:44,782 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:44,782 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:44,783 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:44,783 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,783 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,783 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:44,784 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:44,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:44,784 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:44,784 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:44,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:44,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:44,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:44,784 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:44,785 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:44,785 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:44,785 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:44,785 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:44,785 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:44,785 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:44,785 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:44,785 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:44,785 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:44,785 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2e61d8a0a5c8be60fbc343435192fa13 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-28 14:59:44,785 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=A 2023-04-28 14:59:44,785 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:59:44,785 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=B 2023-04-28 14:59:44,785 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:59:44,785 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=C 2023-04-28 14:59:44,786 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:59:45,119 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/3b56715e0c0946c2a9f819be389a6e90 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/3b56715e0c0946c2a9f819be389a6e90 2023-04-28 14:59:45,124 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2e61d8a0a5c8be60fbc343435192fa13/B of 2e61d8a0a5c8be60fbc343435192fa13 into 3b56715e0c0946c2a9f819be389a6e90(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:45,124 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:45,124 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13., storeName=2e61d8a0a5c8be60fbc343435192fa13/B, priority=13, startTime=1682693984707; duration=0sec 2023-04-28 14:59:45,124 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:45,124 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:59:45,125 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34272 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:59:45,125 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 2e61d8a0a5c8be60fbc343435192fa13/C is initiating minor compaction (all files) 2023-04-28 14:59:45,125 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e61d8a0a5c8be60fbc343435192fa13/C in TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:45,125 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/5392f5a04aab413db26535f541b26e2d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/9f025ae910e349f99b4b92dc145eb726, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/19d3903afe35457097f69e4c326db980] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp, totalSize=33.5 K 2023-04-28 14:59:45,125 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 5392f5a04aab413db26535f541b26e2d, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1682693979767 2023-04-28 14:59:45,125 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 9f025ae910e349f99b4b92dc145eb726, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1682693980341 2023-04-28 14:59:45,125 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 19d3903afe35457097f69e4c326db980, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1682693983061 2023-04-28 14:59:45,129 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 2e61d8a0a5c8be60fbc343435192fa13#C#compaction#471 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:59:45,202 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042866520942fab24a1098ee91e3ee243916 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042866520942fab24a1098ee91e3ee243916 2023-04-28 14:59:45,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:45,247 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:45,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:45,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694045255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:45,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:45,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39972 deadline: 1682694045255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:45,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:45,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694045357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:45,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:45,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39972 deadline: 1682694045358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:45,535 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/32961c0039564e95a60d74a55b789faf as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/32961c0039564e95a60d74a55b789faf 2023-04-28 14:59:45,539 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2e61d8a0a5c8be60fbc343435192fa13/C of 2e61d8a0a5c8be60fbc343435192fa13 into 32961c0039564e95a60d74a55b789faf(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:45,539 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:45,539 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13., storeName=2e61d8a0a5c8be60fbc343435192fa13/C, priority=13, startTime=1682693984707; duration=0sec 2023-04-28 14:59:45,539 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:45,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:45,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39972 deadline: 1682694045560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:45,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:45,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694045561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:45,610 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=387, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/1aa21840d99644f78c2df38663cf9871 2023-04-28 14:59:45,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:45,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694045863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:45,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:45,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39972 deadline: 1682694045864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:45,930 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/66a8815a885c4c66bda12bdae23af7f3 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/66a8815a885c4c66bda12bdae23af7f3 2023-04-28 14:59:45,933 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2e61d8a0a5c8be60fbc343435192fa13/A of 2e61d8a0a5c8be60fbc343435192fa13 into 66a8815a885c4c66bda12bdae23af7f3(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-28 14:59:45,933 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:45,933 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13., storeName=2e61d8a0a5c8be60fbc343435192fa13/A, priority=13, startTime=1682693984707; duration=1sec 2023-04-28 14:59:45,933 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:46,017 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/e771e56486eb4774a68cbb0c285fa615 2023-04-28 14:59:46,025 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/1e7f4d07443f443d887cdf5ccd40844b 2023-04-28 14:59:46,028 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/1aa21840d99644f78c2df38663cf9871 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/1aa21840d99644f78c2df38663cf9871 2023-04-28 14:59:46,031 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/1aa21840d99644f78c2df38663cf9871, entries=150, sequenceid=387, filesize=25.4 K 2023-04-28 14:59:46,032 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/e771e56486eb4774a68cbb0c285fa615 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/e771e56486eb4774a68cbb0c285fa615 2023-04-28 14:59:46,034 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/e771e56486eb4774a68cbb0c285fa615, entries=150, sequenceid=387, filesize=11.9 K 2023-04-28 14:59:46,035 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/1e7f4d07443f443d887cdf5ccd40844b as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/1e7f4d07443f443d887cdf5ccd40844b 2023-04-28 14:59:46,037 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/1e7f4d07443f443d887cdf5ccd40844b, entries=150, sequenceid=387, filesize=11.9 K 2023-04-28 14:59:46,037 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 2e61d8a0a5c8be60fbc343435192fa13 in 1252ms, sequenceid=387, compaction requested=false 2023-04-28 14:59:46,038 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:46,038 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:46,038 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:46,038 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:46,038 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:46,038 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:46,038 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:46,039 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:46,039 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:46,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:46,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:46,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:46,039 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:46,039 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:46,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:46,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:46,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:46,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:46,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:46,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:46,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@934d60f[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:46,040 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:46,040 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:46,041 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:46,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:46,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:46,041 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:46,041 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:46,041 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:46,041 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:46,041 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:46,042 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:46,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:46,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:46,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:46,042 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:46,042 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:46,042 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:46,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:46,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:46,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:46,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:46,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:46,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:46,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:46,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:46,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:46,043 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:46,044 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:46,044 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:46,044 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:46,044 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:46,044 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:46,044 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:46,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:46,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:46,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:46,045 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:46,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:46,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:46,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:46,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:46,045 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:46,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:46,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:46,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:46,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:46,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:46,046 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:46,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:46,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:46,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:46,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:46,047 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:46,047 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:46,047 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:46,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:46,047 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:46,047 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:46,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:46,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:46,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:46,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:46,047 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:46,048 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:46,048 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:46,048 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:46,048 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:46,048 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:46,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:46,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:46,048 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:46,148 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:46,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:46,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:46,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:46,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:46,151 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:46,151 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:46,151 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:46,151 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:46,151 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:46,151 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:46,152 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:46,152 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:46,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:46,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:46,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:46,152 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:46,152 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:46,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:46,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:46,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:46,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:46,153 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:46,153 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:46,153 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:46,153 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:46,153 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:46,153 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:46,153 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:46,153 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:46,153 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:46,153 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2e61d8a0a5c8be60fbc343435192fa13 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 14:59:46,153 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=A 2023-04-28 14:59:46,154 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:46,154 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=B 2023-04-28 14:59:46,154 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:46,154 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=C 2023-04-28 14:59:46,154 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:46,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:46,368 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:46,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693986381, max timestamp=1682693986388Num uniques 859; 2023-04-28 14:59:46,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:59:46,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693986381, max timestamp=1682693986388Num uniques 859; 2023-04-28 14:59:46,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693986381, max timestamp=1682693986388Num uniques 859; 2023-04-28 14:59:46,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:59:46,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:59:46,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:46,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694046407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:46,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:46,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39972 deadline: 1682694046409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:46,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:46,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694046510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:46,513 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:46,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39972 deadline: 1682694046511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:46,570 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428d82014bd92244608bdcac8ee884f9746 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428d82014bd92244608bdcac8ee884f9746 2023-04-28 14:59:46,574 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=400, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/ca95429110584e329a5ea3f613fe6c15 2023-04-28 14:59:46,715 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:46,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39960 deadline: 1682694046713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:46,715 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:46,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39972 deadline: 1682694046713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:46,981 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/acb94e7291484c02a6f0110e7b59da49 2023-04-28 14:59:46,988 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/87899a1935194a4c9c24609222efcd15 2023-04-28 14:59:47,000 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/ca95429110584e329a5ea3f613fe6c15 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/ca95429110584e329a5ea3f613fe6c15 2023-04-28 14:59:47,003 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/ca95429110584e329a5ea3f613fe6c15, entries=150, sequenceid=400, filesize=25.4 K 2023-04-28 14:59:47,003 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/acb94e7291484c02a6f0110e7b59da49 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/acb94e7291484c02a6f0110e7b59da49 2023-04-28 14:59:47,006 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/acb94e7291484c02a6f0110e7b59da49, entries=150, sequenceid=400, filesize=11.9 K 2023-04-28 14:59:47,006 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/87899a1935194a4c9c24609222efcd15 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/87899a1935194a4c9c24609222efcd15 2023-04-28 14:59:47,009 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/87899a1935194a4c9c24609222efcd15, entries=150, sequenceid=400, filesize=11.9 K 2023-04-28 14:59:47,009 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2e61d8a0a5c8be60fbc343435192fa13 in 856ms, sequenceid=400, compaction requested=true 2023-04-28 14:59:47,009 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:47,009 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:47,009 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:47,009 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:47,009 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,009 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:47,009 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:47,010 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,010 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:47,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:47,011 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,011 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:47,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:47,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:47,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:47,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:47,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f8b2353[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:47,012 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:47,012 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,013 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,013 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:47,013 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:47,013 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,013 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:47,013 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:47,014 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,014 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:47,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:47,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:47,014 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:47,014 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:47,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:47,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:47,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:47,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:47,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:47,015 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:47,015 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:47,015 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:47,015 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:47,015 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:47,016 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,016 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:47,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,017 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,017 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:47,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:47,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:47,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:47,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:47,018 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:47,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:47,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:47,019 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2e61d8a0a5c8be60fbc343435192fa13 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-28 14:59:47,019 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,019 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,019 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:47,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:47,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:47,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:47,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:47,019 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:47,019 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:47,020 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=A 2023-04-28 14:59:47,020 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 14:59:47,020 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=B 2023-04-28 14:59:47,020 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 14:59:47,020 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=C 2023-04-28 14:59:47,021 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 14:59:47,019 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,019 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:47,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:47,022 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:47,021 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,022 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,022 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,022 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:47,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:47,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:47,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39972 deadline: 1682694047040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:47,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39960 deadline: 1682694047040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,122 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:47,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:47,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:47,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:47,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:47,124 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:47,124 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:47,124 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:47,124 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:47,125 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,125 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,126 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:47,126 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:47,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:47,126 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,126 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:47,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:47,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:47,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:47,127 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:47,127 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:47,127 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:47,127 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:47,127 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:47,127 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:47,127 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:47,127 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:47,127 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:47,127 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:47,127 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:47,127 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:47,127 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,127 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:47,127 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:47,128 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:47,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:47,128 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:47,128 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,128 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:47,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:47,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:47,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:47,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39f4ccbf[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:47,129 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:47,129 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,129 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,129 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:47,129 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:47,129 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:47,130 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,130 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:47,131 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,131 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:47,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:47,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:47,131 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:47,131 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:47,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:47,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:47,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:47,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:47,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:47,133 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:47,133 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:47,133 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:47,133 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:47,133 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:47,134 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:47,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:47,134 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,134 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:47,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:47,134 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:47,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:47,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:47,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,135 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:47,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,136 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,136 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,136 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:47,136 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,137 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:47,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:47,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:47,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:47,137 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:47,137 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:47,137 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,137 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,137 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,137 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:47,137 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:47,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:47,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:47,145 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:47,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39960 deadline: 1682694047143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,146 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:47,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39972 deadline: 1682694047143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,237 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:47,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:47,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:47,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:47,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:47,240 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:47,240 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:47,240 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:47,240 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:47,240 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,241 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,241 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,241 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:47,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:47,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:47,242 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,242 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:47,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:47,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,242 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:47,242 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:47,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:47,243 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:47,243 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:47,243 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:47,243 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:47,243 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:47,243 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:47,243 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:47,243 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:47,243 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:47,243 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:47,243 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:47,243 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,243 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:47,243 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:47,244 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,244 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:47,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:47,244 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,244 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:47,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:47,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:47,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:47,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:47,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@86721bd[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:47,245 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:47,245 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,246 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,246 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:47,246 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:47,246 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:47,246 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,246 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:47,246 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:47,246 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,246 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:47,247 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:47,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:47,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:47,248 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:47,248 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:47,248 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:47,248 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:47,248 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:47,249 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,249 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:47,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:47,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,249 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:47,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:47,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:47,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:47,250 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:47,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,250 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:47,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,251 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,251 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:47,251 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,252 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:47,252 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:47,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:47,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:47,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:47,252 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:47,252 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:47,252 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:47,252 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:47,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:47,252 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,252 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,252 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:47,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39972 deadline: 1682694047346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:47,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39960 deadline: 1682694047347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,352 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:47,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:47,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:47,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:47,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:47,356 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:47,356 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:47,356 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:47,356 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:47,356 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,357 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,357 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:47,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:47,357 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:47,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:47,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,359 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:47,359 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,359 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:47,362 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:47,362 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:47,362 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:47,362 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:47,362 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:47,374 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:47,374 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:47,374 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:47,374 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:47,374 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:47,374 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:47,374 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:47,374 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:47,374 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,374 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:47,374 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:47,376 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:47,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:47,376 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:47,377 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,377 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:47,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:47,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:47,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:47,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@274c1a10[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:47,378 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:47,378 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,379 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,379 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:47,379 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,379 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:47,379 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:47,379 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:47,380 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:47,380 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:47,380 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:47,380 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:47,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:47,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:47,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:47,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:47,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:47,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:47,382 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:47,382 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:47,382 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:47,382 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:47,382 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:47,383 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,383 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:47,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:47,383 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:47,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:47,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:47,384 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:47,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:47,385 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:47,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,387 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,387 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:47,388 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:47,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:47,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:47,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:47,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:47,388 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:47,387 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,388 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,388 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,388 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,388 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,388 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,388 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:47,388 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:47,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:47,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:47,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:47,434 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304282f9816b714324c1ba7bfa52e115bebc5 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304282f9816b714324c1ba7bfa52e115bebc5 2023-04-28 14:59:47,488 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:47,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:47,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:47,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:47,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:47,491 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:47,491 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:47,491 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:47,491 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:47,492 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,492 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,493 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:47,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:47,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:47,493 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,493 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,493 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:47,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:47,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,494 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:47,494 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:47,494 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:47,494 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:47,494 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:47,494 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:47,494 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:47,494 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:47,494 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:47,494 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:47,494 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:47,494 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:47,494 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:47,494 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:47,494 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,495 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:47,495 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:47,495 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,495 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:47,496 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:47,496 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:47,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:47,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51ab3411[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:47,497 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:47,497 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,498 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,498 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:47,498 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:47,498 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:47,498 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,498 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:47,498 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:47,498 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:47,499 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:47,499 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:47,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:47,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:47,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:47,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:47,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:47,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:47,501 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:47,501 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:47,501 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:47,501 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:47,501 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:47,503 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,503 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,503 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:47,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:47,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:47,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,504 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:47,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:47,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:47,504 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:47,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:47,506 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,506 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,506 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,506 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,506 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,506 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:47,507 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:47,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:47,507 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:47,507 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:47,507 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:47,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:47,507 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:47,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:47,507 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:47,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:47,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:47,607 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:47,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:47,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:47,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:47,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:47,610 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:47,610 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:47,610 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:47,610 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:47,610 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,610 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,611 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,611 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:47,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:47,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:47,611 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,611 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:47,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:47,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,612 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:47,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:47,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:47,612 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:47,612 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:47,612 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:47,612 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:47,612 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:47,612 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:47,612 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:47,612 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:47,612 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:47,612 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:47,612 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:47,612 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,612 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:47,612 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:47,613 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,613 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:47,613 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,613 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:47,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:47,615 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:47,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e259baf[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:47,615 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,615 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,615 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:47,615 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:47,615 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:47,615 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,615 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:47,616 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:47,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:47,616 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:47,616 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:47,616 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:47,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:47,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:47,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:47,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:47,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:47,618 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:47,618 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:47,618 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:47,618 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:47,618 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:47,619 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:47,619 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,619 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:47,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:47,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:47,619 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:47,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:47,621 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,622 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,622 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,622 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,622 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,622 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,622 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,622 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,622 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:47,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:47,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:47,622 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:47,622 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:47,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:47,622 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:47,622 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:47,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:47,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:47,652 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:47,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39972 deadline: 1682694047651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:47,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39960 deadline: 1682694047653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,722 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:47,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:47,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:47,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:47,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:47,725 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:47,725 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:47,725 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:47,725 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:47,725 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,725 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,726 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,726 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:47,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:47,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:47,726 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,726 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:47,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:47,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,727 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:47,727 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:47,727 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:47,727 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:47,727 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:47,727 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:47,727 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:47,727 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:47,727 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:47,727 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:47,727 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:47,727 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:47,727 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:47,727 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:47,727 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,728 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:47,728 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:47,728 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,728 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:47,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:47,728 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,728 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:47,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:47,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:47,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:47,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:47,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ffe311f[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:47,730 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:47,730 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,730 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,730 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:47,730 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:47,730 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:47,730 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,730 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:47,731 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,731 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:47,731 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:47,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:47,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:47,731 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:47,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:47,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:47,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:47,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:47,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:47,733 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:47,733 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:47,733 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:47,733 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:47,733 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:47,734 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:47,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:47,734 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,735 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:47,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:47,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:47,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:47,735 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:47,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,736 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:47,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,737 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,737 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,737 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,737 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:47,737 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,737 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:47,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:47,737 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:47,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:47,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:47,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:47,737 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,738 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,738 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:47,738 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:47,738 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:47,838 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:47,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:47,839 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=438, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/3591ca20f8374d62993aa4b1c08e5b9a 2023-04-28 14:59:47,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:47,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:47,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:47,841 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:47,841 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:47,841 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:47,841 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:47,841 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,841 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,842 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,842 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:47,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:47,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:47,842 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,842 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:47,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:47,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:47,843 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:47,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:47,844 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:47,844 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:47,844 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:47,844 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:47,844 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:47,844 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:47,844 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:47,844 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:47,844 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:47,844 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:47,844 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:47,844 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,844 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:47,844 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:47,846 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,846 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:47,846 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,846 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:47,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:47,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:47,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3825286a[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:47,847 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:47,848 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,848 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,848 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:47,848 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:47,848 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:47,848 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,848 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:47,849 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:47,849 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:47,849 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:47,849 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:47,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:47,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,851 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/898142dcaab04e3dbf1fd9c2d6ee2f23 2023-04-28 14:59:47,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:47,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:47,851 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:47,852 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:47,852 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:47,852 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:47,852 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:47,853 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,853 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,853 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:47,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:47,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:47,853 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:47,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:47,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:47,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:47,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,854 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:47,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,859 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,859 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:47,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:47,859 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:47,859 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,859 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:47,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:47,860 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:47,860 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:47,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:47,860 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,860 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:47,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:47,860 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,860 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,860 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:47,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:47,960 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:47,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:47,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:47,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:47,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:47,962 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:47,962 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:47,962 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:47,962 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:47,963 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,963 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,963 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:47,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:47,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:47,963 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,965 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,965 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:47,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:47,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,966 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:47,966 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:47,966 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:47,966 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:47,966 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:47,966 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:47,966 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:47,966 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:47,966 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:47,966 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:47,967 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:47,967 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:47,967 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:47,967 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:47,967 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,967 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:47,967 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:47,967 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,968 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:47,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:47,968 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:47,968 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:47,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:47,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:47,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:47,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@302688c3[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:47,969 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:47,969 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,969 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,970 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:47,970 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:47,970 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:47,970 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,970 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:47,971 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:47,971 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,971 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:47,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:47,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:47,971 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:47,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:47,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:47,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:47,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:47,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:47,973 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:47,973 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:47,973 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:47,973 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:47,973 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:47,973 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,973 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:47,973 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:47,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,974 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:47,975 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:47,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:47,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:47,977 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:47,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:47,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:47,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:47,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:47,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:47,977 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:47,977 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:47,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:47,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:47,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:47,977 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:47,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:47,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:48,077 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:48,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:48,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:48,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:48,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:48,080 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:48,080 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:48,080 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:48,080 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:48,080 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:48,080 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:48,081 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,081 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:48,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:48,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:48,081 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,081 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:48,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:48,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:48,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:48,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:48,082 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:48,082 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:48,082 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:48,082 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:48,082 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:48,082 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:48,082 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:48,082 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:48,082 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:48,082 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:48,082 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:48,083 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:48,083 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:48,083 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:48,083 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,083 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:48,083 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:48,083 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:48,083 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:48,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:48,084 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:48,084 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:48,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:48,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:48,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:48,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:48,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:48,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52bc858b[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:48,085 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:48,085 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:48,085 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:48,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:48,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:48,085 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:48,086 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:48,086 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:48,086 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,086 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:48,086 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,086 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:48,086 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:48,086 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:48,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:48,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:48,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:48,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:48,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:48,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:48,088 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:48,088 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:48,088 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:48,088 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:48,088 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:48,089 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:48,089 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:48,089 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:48,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:48,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:48,089 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:48,090 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,092 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:48,092 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:48,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:48,092 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:48,092 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:48,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:48,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:48,092 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:48,092 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:48,092 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:48,092 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:48,092 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:48,092 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:48,092 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:48,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:48,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:48,092 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:48,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:48,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39972 deadline: 1682694048153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,157 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:48,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39960 deadline: 1682694048155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,192 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:48,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:48,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:48,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:48,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:48,195 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:48,195 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:48,195 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:48,195 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:48,195 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:48,195 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:48,196 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,196 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:48,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:48,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:48,196 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,196 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:48,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:48,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:48,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:48,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:48,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:48,197 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:48,197 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:48,197 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:48,197 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:48,197 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:48,197 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:48,197 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:48,197 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:48,197 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:48,197 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:48,198 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:48,198 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:48,198 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:48,198 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,198 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:48,198 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:48,198 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:48,198 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:48,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:48,199 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:48,199 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:48,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:48,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:48,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:48,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:48,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:48,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cc4200a[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:48,200 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:48,200 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:48,200 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:48,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:48,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:48,201 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:48,201 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:48,201 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:48,201 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,201 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:48,201 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:48,201 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:48,201 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,201 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:48,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:48,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:48,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:48,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:48,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:48,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:48,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:48,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:48,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:48,203 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:48,203 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:48,203 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:48,203 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:48,203 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:48,205 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:48,205 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:48,205 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:48,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:48,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:48,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:48,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:48,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:48,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:48,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:48,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:48,205 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:48,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:48,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:48,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:48,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,206 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:48,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:48,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:48,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:48,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:48,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:48,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:48,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:48,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:48,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:48,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:48,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:48,208 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:48,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:48,208 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:48,209 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:48,263 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/1f7ef88a5cc44c218bab8728e48d1244 2023-04-28 14:59:48,265 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/3591ca20f8374d62993aa4b1c08e5b9a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/3591ca20f8374d62993aa4b1c08e5b9a 2023-04-28 14:59:48,268 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/3591ca20f8374d62993aa4b1c08e5b9a, entries=250, sequenceid=438, filesize=39.2 K 2023-04-28 14:59:48,268 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/898142dcaab04e3dbf1fd9c2d6ee2f23 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/898142dcaab04e3dbf1fd9c2d6ee2f23 2023-04-28 14:59:48,271 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/898142dcaab04e3dbf1fd9c2d6ee2f23, entries=150, sequenceid=438, filesize=11.9 K 2023-04-28 14:59:48,271 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/1f7ef88a5cc44c218bab8728e48d1244 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/1f7ef88a5cc44c218bab8728e48d1244 2023-04-28 14:59:48,274 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/1f7ef88a5cc44c218bab8728e48d1244, entries=150, sequenceid=438, filesize=11.9 K 2023-04-28 14:59:48,275 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~420.80 KB/430896, currentSize=33.54 KB/34350 for 2e61d8a0a5c8be60fbc343435192fa13 in 1256ms, sequenceid=438, compaction requested=true 2023-04-28 14:59:48,275 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:48,275 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:48,275 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:59:48,275 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:59:48,275 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:48,275 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:48,276 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 118678 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:59:48,276 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49154 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:59:48,276 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): 2e61d8a0a5c8be60fbc343435192fa13/A is initiating minor compaction (all files) 2023-04-28 14:59:48,276 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 2e61d8a0a5c8be60fbc343435192fa13/B is initiating minor compaction (all files) 2023-04-28 14:59:48,276 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e61d8a0a5c8be60fbc343435192fa13/A in TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:48,276 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e61d8a0a5c8be60fbc343435192fa13/B in TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:48,276 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/66a8815a885c4c66bda12bdae23af7f3, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/1aa21840d99644f78c2df38663cf9871, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/ca95429110584e329a5ea3f613fe6c15, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/3591ca20f8374d62993aa4b1c08e5b9a] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp, totalSize=115.9 K 2023-04-28 14:59:48,276 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/3b56715e0c0946c2a9f819be389a6e90, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/e771e56486eb4774a68cbb0c285fa615, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/acb94e7291484c02a6f0110e7b59da49, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/898142dcaab04e3dbf1fd9c2d6ee2f23] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp, totalSize=48.0 K 2023-04-28 14:59:48,277 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 66a8815a885c4c66bda12bdae23af7f3, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1682693980344 2023-04-28 14:59:48,277 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 3b56715e0c0946c2a9f819be389a6e90, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1682693980344 2023-04-28 14:59:48,277 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 1aa21840d99644f78c2df38663cf9871, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1682693983115 2023-04-28 14:59:48,277 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting e771e56486eb4774a68cbb0c285fa615, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1682693983115 2023-04-28 14:59:48,277 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting ca95429110584e329a5ea3f613fe6c15, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1682693985251 2023-04-28 14:59:48,278 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting acb94e7291484c02a6f0110e7b59da49, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1682693985251 2023-04-28 14:59:48,278 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 3591ca20f8374d62993aa4b1c08e5b9a, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682693986400 2023-04-28 14:59:48,278 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 898142dcaab04e3dbf1fd9c2d6ee2f23, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682693986405 2023-04-28 14:59:48,291 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 2e61d8a0a5c8be60fbc343435192fa13#B#compaction#481 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:59:48,309 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:48,309 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2e61d8a0a5c8be60fbc343435192fa13#A#compaction#480 average throughput is 1.04 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:59:48,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:48,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:48,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:48,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:48,320 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:48,321 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:48,321 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:48,321 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:48,321 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:48,321 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:48,322 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:48,322 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:48,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:48,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:48,322 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,322 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:48,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:48,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:48,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:48,323 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:48,338 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:48,338 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:48,338 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:48,338 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:48,338 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:48,338 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:48,338 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:48,338 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:48,338 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:48,338 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2e61d8a0a5c8be60fbc343435192fa13 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-28 14:59:48,339 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=A 2023-04-28 14:59:48,339 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:48,339 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=B 2023-04-28 14:59:48,339 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:48,339 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=C 2023-04-28 14:59:48,340 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:48,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:48,502 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:48,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682693988509, max timestamp=1682693988534Num uniques 859; 2023-04-28 14:59:48,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 14:59:48,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682693988509, max timestamp=1682693988534Num uniques 859; 2023-04-28 14:59:48,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682693988509, max timestamp=1682693988534Num uniques 859; 2023-04-28 14:59:48,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 14:59:48,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 14:59:48,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:48,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39948 deadline: 1682694048551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:48,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39988 deadline: 1682694048554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,559 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:48,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:40006 deadline: 1682694048555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:48,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:48,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39948 deadline: 1682694048656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39988 deadline: 1682694048657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:48,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:40006 deadline: 1682694048660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,712 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/7036e801e15e4a648b3347159b70de5d as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/7036e801e15e4a648b3347159b70de5d 2023-04-28 14:59:48,716 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2e61d8a0a5c8be60fbc343435192fa13/B of 2e61d8a0a5c8be60fbc343435192fa13 into 7036e801e15e4a648b3347159b70de5d(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:48,716 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:48,716 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13., storeName=2e61d8a0a5c8be60fbc343435192fa13/B, priority=12, startTime=1682693988275; duration=0sec 2023-04-28 14:59:48,716 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:48,716 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 14:59:48,717 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49154 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 14:59:48,717 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 2e61d8a0a5c8be60fbc343435192fa13/C is initiating minor compaction (all files) 2023-04-28 14:59:48,717 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e61d8a0a5c8be60fbc343435192fa13/C in TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:48,717 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/32961c0039564e95a60d74a55b789faf, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/1e7f4d07443f443d887cdf5ccd40844b, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/87899a1935194a4c9c24609222efcd15, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/1f7ef88a5cc44c218bab8728e48d1244] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp, totalSize=48.0 K 2023-04-28 14:59:48,718 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 32961c0039564e95a60d74a55b789faf, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1682693980344 2023-04-28 14:59:48,718 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 1e7f4d07443f443d887cdf5ccd40844b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1682693983115 2023-04-28 14:59:48,718 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 87899a1935194a4c9c24609222efcd15, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1682693985251 2023-04-28 14:59:48,718 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 1f7ef88a5cc44c218bab8728e48d1244, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682693986405 2023-04-28 14:59:48,722 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 2e61d8a0a5c8be60fbc343435192fa13#C#compaction#483 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:59:48,758 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/989d9db29d234a7bab2e46834ef13b30 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/989d9db29d234a7bab2e46834ef13b30 2023-04-28 14:59:48,759 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304281ba1283d4fb64d38a9feedc378c235ef to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304281ba1283d4fb64d38a9feedc378c235ef 2023-04-28 14:59:48,762 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2e61d8a0a5c8be60fbc343435192fa13/A of 2e61d8a0a5c8be60fbc343435192fa13 into 989d9db29d234a7bab2e46834ef13b30(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:48,762 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:48,762 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13., storeName=2e61d8a0a5c8be60fbc343435192fa13/A, priority=12, startTime=1682693988275; duration=0sec 2023-04-28 14:59:48,762 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:48,764 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=446, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/3f019e3e5f9346319065db6dff0a2cff 2023-04-28 14:59:48,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:48,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39988 deadline: 1682694048860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:48,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39948 deadline: 1682694048861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:48,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:48,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:40006 deadline: 1682694048866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,129 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/b2a3a124263143b2b1088e65493da976 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/b2a3a124263143b2b1088e65493da976 2023-04-28 14:59:49,131 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2e61d8a0a5c8be60fbc343435192fa13/C of 2e61d8a0a5c8be60fbc343435192fa13 into b2a3a124263143b2b1088e65493da976(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:49,132 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:49,132 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13., storeName=2e61d8a0a5c8be60fbc343435192fa13/C, priority=12, startTime=1682693988275; duration=0sec 2023-04-28 14:59:49,132 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:49,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:49,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39988 deadline: 1682694049162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,166 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:49,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39972 deadline: 1682694049164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,167 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:49,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39960 deadline: 1682694049166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,167 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:49,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39948 deadline: 1682694049166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:49,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:40006 deadline: 1682694049168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,174 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/5da7fdde2e1b4e2ba8e553a09924a365 2023-04-28 14:59:49,585 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/2b59847a41794c7590d6d027a9b0ba79 2023-04-28 14:59:49,588 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/3f019e3e5f9346319065db6dff0a2cff as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/3f019e3e5f9346319065db6dff0a2cff 2023-04-28 14:59:49,590 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/3f019e3e5f9346319065db6dff0a2cff, entries=100, sequenceid=446, filesize=18.6 K 2023-04-28 14:59:49,591 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/5da7fdde2e1b4e2ba8e553a09924a365 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/5da7fdde2e1b4e2ba8e553a09924a365 2023-04-28 14:59:49,593 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/5da7fdde2e1b4e2ba8e553a09924a365, entries=100, sequenceid=446, filesize=9.5 K 2023-04-28 14:59:49,594 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/2b59847a41794c7590d6d027a9b0ba79 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/2b59847a41794c7590d6d027a9b0ba79 2023-04-28 14:59:49,596 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/2b59847a41794c7590d6d027a9b0ba79, entries=100, sequenceid=446, filesize=9.5 K 2023-04-28 14:59:49,596 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 2e61d8a0a5c8be60fbc343435192fa13 in 1258ms, sequenceid=446, compaction requested=false 2023-04-28 14:59:49,596 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:49,596 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:49,596 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:49,597 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:49,597 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,597 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:49,597 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:49,597 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,597 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:49,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:49,598 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,598 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:49,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:49,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:49,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:49,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:49,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:49,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d332580[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:49,599 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:49,599 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,606 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,607 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:49,607 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:49,607 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:49,607 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,607 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:49,608 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:49,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:49,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:49,608 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:49,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:49,608 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:49,608 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:49,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:49,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:49,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:49,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:49,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:49,609 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:49,609 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:49,609 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:49,609 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:49,609 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:49,610 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,610 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,610 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:49,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:49,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:49,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:49,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:49,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:49,611 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:49,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:49,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:49,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,612 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:49,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:49,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,613 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,613 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:49,614 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:49,614 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,614 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:49,614 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:49,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:49,614 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:49,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:49,614 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:49,614 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:49,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:49,614 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,614 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:49,614 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:49,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:49,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:49,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682693988550, max timestamp=1682693989670Num uniques 1118; 2023-04-28 14:59:49,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-28 14:59:49,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682693988550, max timestamp=1682693989670Num uniques 1118; 2023-04-28 14:59:49,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-28 14:59:49,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682693988550, max timestamp=1682693989670Num uniques 1118; 2023-04-28 14:59:49,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:49,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-28 14:59:49,673 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2e61d8a0a5c8be60fbc343435192fa13 3/3 column families, dataSize=26.84 KB heapSize=65.39 KB 2023-04-28 14:59:49,673 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=A 2023-04-28 14:59:49,673 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:59:49,673 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=B 2023-04-28 14:59:49,673 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:59:49,674 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=C 2023-04-28 14:59:49,674 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:59:49,683 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042816cfb0a12a94495ca05b1960e08f6fd5 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042816cfb0a12a94495ca05b1960e08f6fd5 2023-04-28 14:59:49,692 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693989687, max timestamp=1682693989691Num uniques 859; 2023-04-28 14:59:49,692 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:59:49,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693989687, max timestamp=1682693989691Num uniques 859; 2023-04-28 14:59:49,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682693989687, max timestamp=1682693989691Num uniques 859; 2023-04-28 14:59:49,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:59:49,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 14:59:49,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682693989705, max timestamp=1682693989709Num uniques 1018; 2023-04-28 14:59:49,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-28 14:59:49,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682693989705, max timestamp=1682693989709Num uniques 1018; 2023-04-28 14:59:49,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-28 14:59:49,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682693989705, max timestamp=1682693989709Num uniques 1018; 2023-04-28 14:59:49,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-28 14:59:49,714 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:49,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:49,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:49,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:49,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:49,717 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:49,717 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:49,717 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:49,717 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:49,717 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,717 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:49,718 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:49,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:49,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:49,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:49,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:49,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:49,719 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:49,719 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,720 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,720 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:49,720 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:49,720 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:49,720 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:49,720 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:49,720 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:49,721 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:49,721 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:49,721 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:49,721 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:49,721 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:49,721 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:49,721 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:49,721 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:49,721 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,721 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:49,721 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:49,722 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:49,722 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,723 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,723 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:49,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:49,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:49,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:49,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:49,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:49,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49d7dc75[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:49,724 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:49,724 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,725 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,725 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:49,725 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:49,725 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:49,725 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,725 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:49,726 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,726 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:49,726 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:49,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:49,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:49,726 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:49,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:49,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:49,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:49,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:49,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:49,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:49,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:49,728 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:49,729 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:49,729 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:49,729 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:49,729 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:49,730 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,730 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:49,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:49,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,730 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:49,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:49,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:49,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:49,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,731 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:49,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:49,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:49,732 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:49,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:49,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,743 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,743 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:49,743 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:49,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:49,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:49,743 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:49,743 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,743 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,743 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,743 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:49,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:49,743 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:49,744 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:49,743 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:49,744 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:49,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:49,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:49,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:49,746 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:49,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39988 deadline: 1682694049742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8092, totalHeapSize=18648, min timestamp=1682693989730, max timestamp=1682693989745Num uniques 1127; 2023-04-28 14:59:49,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=51602, new segment data size=8092, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-28 14:59:49,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8093, totalHeapSize=18648, min timestamp=1682693989730, max timestamp=1682693989745Num uniques 1127; 2023-04-28 14:59:49,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=51603, new segment data size=8093, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-28 14:59:49,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8093, totalHeapSize=18648, min timestamp=1682693989730, max timestamp=1682693989745Num uniques 1127; 2023-04-28 14:59:49,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=51603, new segment data size=8093, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-28 14:59:49,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682693989756, max timestamp=1682693989769Num uniques 1036; 2023-04-28 14:59:49,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-28 14:59:49,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682693989756, max timestamp=1682693989769Num uniques 1036; 2023-04-28 14:59:49,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682693989756, max timestamp=1682693989769Num uniques 1036; 2023-04-28 14:59:49,772 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-28 14:59:49,772 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-28 14:59:49,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682693989779, max timestamp=1682693989796Num uniques 1095; 2023-04-28 14:59:49,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-28 14:59:49,799 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682693989779, max timestamp=1682693989796Num uniques 1095; 2023-04-28 14:59:49,799 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-28 14:59:49,799 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682693989779, max timestamp=1682693989796Num uniques 1095; 2023-04-28 14:59:49,799 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-28 14:59:49,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9334, totalHeapSize=21456, min timestamp=1682693989811, max timestamp=1682693989823Num uniques 1104; 2023-04-28 14:59:49,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9339, totalHeapSize=21456, min timestamp=1682693989811, max timestamp=1682693989823Num uniques 1104; 2023-04-28 14:59:49,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=50554, new segment data size=9334, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-28 14:59:49,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9334, totalHeapSize=21456, min timestamp=1682693989811, max timestamp=1682693989823Num uniques 1104; 2023-04-28 14:59:49,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=50559, new segment data size=9339, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-28 14:59:49,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=50554, new segment data size=9334, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-28 14:59:49,844 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:49,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:49,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:49,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:49,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:49,846 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:49,846 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:49,846 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:49,846 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:49,847 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,847 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:49,847 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,847 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:49,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:49,848 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,848 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:49,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:49,848 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:49,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:49,849 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:49,849 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:49,849 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:49,849 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:49,849 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:49,849 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:49,849 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:49,849 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:49,849 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:49,849 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:49,849 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:49,849 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,849 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:49,849 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:49,850 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:49,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:49,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:49,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:49,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:49,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:49,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1682693989831, max timestamp=1682693989849Num uniques 1063; 2023-04-28 14:59:49,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-04-28 14:59:49,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1682693989831, max timestamp=1682693989849Num uniques 1063; 2023-04-28 14:59:49,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-04-28 14:59:49,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1682693989831, max timestamp=1682693989849Num uniques 1063; 2023-04-28 14:59:49,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-04-28 14:59:49,852 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:49,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f718819[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:49,852 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,852 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:49,852 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:49,852 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,853 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,853 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:49,853 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:49,853 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:49,853 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,853 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:49,854 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,854 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:49,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:49,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:49,854 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:49,854 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:49,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:49,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:49,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:49,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:49,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:49,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:49,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:49,855 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:49,855 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:49,855 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:49,855 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:49,855 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:49,856 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:49,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:49,856 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,856 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:49,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:49,857 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:49,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,858 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:49,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:49,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,859 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,859 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:49,859 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:49,859 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:49,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:49,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:49,859 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:49,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:49,859 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:49,859 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:49,860 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:49,859 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:49,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:49,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:49,859 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,860 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,860 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:49,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:40006 deadline: 1682694049871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682693989867, max timestamp=1682693989871Num uniques 1122; 2023-04-28 14:59:49,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=10162, suffix heap size=144656,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=222 2023-04-28 14:59:49,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682693989867, max timestamp=1682693989871Num uniques 1122; 2023-04-28 14:59:49,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=10162, suffix heap size=144656,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=222 2023-04-28 14:59:49,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682693989867, max timestamp=1682693989871Num uniques 1122; 2023-04-28 14:59:49,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=7872, suffix heap size=144656,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=172 2023-04-28 14:59:49,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682693989880, max timestamp=1682693989894Num uniques 1081; 2023-04-28 14:59:49,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-28 14:59:49,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682693989880, max timestamp=1682693989894Num uniques 1081; 2023-04-28 14:59:49,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682693989880, max timestamp=1682693989894Num uniques 1031; 2023-04-28 14:59:49,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-28 14:59:49,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-28 14:59:49,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1682693989912, max timestamp=1682693989923Num uniques 1090; 2023-04-28 14:59:49,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1682693989912, max timestamp=1682693989923Num uniques 1090; 2023-04-28 14:59:49,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1682693989912, max timestamp=1682693989923Num uniques 1090; 2023-04-28 14:59:49,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-28 14:59:49,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-28 14:59:49,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-28 14:59:49,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1682693989942, max timestamp=1682693989951Num uniques 1099; 2023-04-28 14:59:49,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-28 14:59:49,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1682693989942, max timestamp=1682693989951Num uniques 1099; 2023-04-28 14:59:49,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-28 14:59:49,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1682693989942, max timestamp=1682693989951Num uniques 1099; 2023-04-28 14:59:49,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-28 14:59:49,960 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:49,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:49,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:49,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:49,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:49,962 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:49,962 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:49,962 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:49,962 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:49,962 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,962 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:49,963 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,963 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:49,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:49,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:49,963 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,963 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:49,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:49,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:49,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:49,964 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:49,964 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:49,964 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:49,964 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:49,964 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:49,964 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:49,964 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:49,964 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:49,964 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:49,964 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:49,964 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:49,964 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:49,964 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:49,964 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:49,964 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,964 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:49,964 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:49,966 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,966 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:49,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:49,966 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,966 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:49,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:49,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:49,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:49,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:49,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:49,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@248238d[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:49,967 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:49,967 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,968 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,968 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:49,968 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:49,968 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:49,968 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,968 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:49,969 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:49,969 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:49,969 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,969 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:49,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:49,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:49,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:49,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:49,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:49,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:49,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:49,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:49,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:49,971 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:49,971 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:49,971 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:49,971 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:49,971 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:49,972 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,972 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:49,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:49,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,972 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:49,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:49,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:49,973 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:49,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:49,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:49,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:49,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,974 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:49,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:49,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,976 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,976 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:49,976 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:49,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:49,976 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:49,976 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:49,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:49,976 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:49,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:49,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:49,976 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:49,976 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,976 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:49,976 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:49,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:49,976 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:49,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:49,976 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:49,982 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1682693989972, max timestamp=1682693989981Num uniques 1108; 2023-04-28 14:59:49,982 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-04-28 14:59:49,983 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1682693989972, max timestamp=1682693989981Num uniques 1108; 2023-04-28 14:59:49,983 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-04-28 14:59:49,983 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1682693989972, max timestamp=1682693989981Num uniques 1108; 2023-04-28 14:59:49,983 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-04-28 14:59:50,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682693989993, max timestamp=1682693990000Num uniques 1017; 2023-04-28 14:59:50,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-28 14:59:50,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682693989993, max timestamp=1682693990000Num uniques 1017; 2023-04-28 14:59:50,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-28 14:59:50,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682693989993, max timestamp=1682693990000Num uniques 1017; 2023-04-28 14:59:50,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-28 14:59:50,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1682693990007, max timestamp=1682693990017Num uniques 1076; 2023-04-28 14:59:50,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=10346, suffix heap size=139872,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=226 2023-04-28 14:59:50,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1682693990007, max timestamp=1682693990017Num uniques 1076; 2023-04-28 14:59:50,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=10346, suffix heap size=139872,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=226 2023-04-28 14:59:50,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10345, totalHeapSize=23744, min timestamp=1682693990007, max timestamp=1682693990018Num uniques 1076; 2023-04-28 14:59:50,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=49275, new segment data size=10345, suffix heap size=139872,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=226 2023-04-28 14:59:50,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1682693990026, max timestamp=1682693990035Num uniques 1085; 2023-04-28 14:59:50,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-04-28 14:59:50,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1682693990026, max timestamp=1682693990035Num uniques 1085; 2023-04-28 14:59:50,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-04-28 14:59:50,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1682693990026, max timestamp=1682693990035Num uniques 1085; 2023-04-28 14:59:50,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-04-28 14:59:50,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1682693990046, max timestamp=1682693990053Num uniques 1094; 2023-04-28 14:59:50,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=11174, suffix heap size=141744,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=244 2023-04-28 14:59:50,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1682693990046, max timestamp=1682693990053Num uniques 1094; 2023-04-28 14:59:50,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=11174, suffix heap size=141744,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=244 2023-04-28 14:59:50,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1682693990046, max timestamp=1682693990053Num uniques 1094; 2023-04-28 14:59:50,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=11174, suffix heap size=141744,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=244 2023-04-28 14:59:50,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9288, totalHeapSize=21352, min timestamp=1682693990066, max timestamp=1682693990070Num uniques 1103; 2023-04-28 14:59:50,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=50508, new segment data size=9288, suffix heap size=142680,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=203 2023-04-28 14:59:50,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9288, totalHeapSize=21352, min timestamp=1682693990066, max timestamp=1682693990070Num uniques 1103; 2023-04-28 14:59:50,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=50508, new segment data size=9288, suffix heap size=142680,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=203 2023-04-28 14:59:50,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9288, totalHeapSize=21352, min timestamp=1682693990066, max timestamp=1682693990070Num uniques 1103; 2023-04-28 14:59:50,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=50508, new segment data size=9288, suffix heap size=142680,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=203 2023-04-28 14:59:50,077 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:50,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:50,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:50,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:50,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:50,079 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:50,079 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:50,079 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:50,079 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:50,080 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,080 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:50,080 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:50,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:50,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:50,080 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:50,081 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,081 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:50,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:50,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:50,081 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:50,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:50,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:50,081 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:50,081 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:50,081 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:50,081 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:50,081 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:50,081 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:50,081 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:50,081 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:50,081 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:50,082 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:50,082 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:50,082 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,082 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:50,082 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:50,082 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,082 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:50,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:50,083 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,083 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:50,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:50,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:50,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:50,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:50,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:50,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56831961[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:50,084 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:50,084 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,084 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,084 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:50,084 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,085 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:50,084 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:50,085 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:50,085 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:50,085 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:50,085 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,085 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:50,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:50,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:50,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:50,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:50,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:50,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:50,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:50,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:50,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:50,087 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:50,087 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:50,087 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:50,087 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:50,087 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:50,088 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=487, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/f4c341c4701a4db1a14923d04b5bd15e 2023-04-28 14:59:50,088 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,088 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:50,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:50,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,088 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:50,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:50,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:50,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:50,089 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:50,090 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:50,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:50,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:50,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:50,092 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,092 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:50,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:50,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:50,092 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:50,092 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:50,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:50,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:50,092 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:50,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:50,092 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:50,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:50,092 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,092 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,092 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,092 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:50,092 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:50,093 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:50,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1682693990085, max timestamp=1682693990094Num uniques 1062; 2023-04-28 14:59:50,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=7412, suffix heap size=138416,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=162 2023-04-28 14:59:50,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1682693990085, max timestamp=1682693990094Num uniques 1062; 2023-04-28 14:59:50,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=7412, suffix heap size=138416,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=162 2023-04-28 14:59:50,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1682693990085, max timestamp=1682693990094Num uniques 1062; 2023-04-28 14:59:50,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=7412, suffix heap size=138416,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=162 2023-04-28 14:59:50,107 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=487 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/b4f2253a79684a679d741ffe1448f2c7 2023-04-28 14:59:50,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1682693990105, max timestamp=1682693990113Num uniques 1021; 2023-04-28 14:59:50,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=12406, suffix heap size=134152,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=271 2023-04-28 14:59:50,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1682693990105, max timestamp=1682693990113Num uniques 1021; 2023-04-28 14:59:50,116 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=12406, suffix heap size=134152,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=271 2023-04-28 14:59:50,116 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1682693990108, max timestamp=1682693990113Num uniques 1021; 2023-04-28 14:59:50,116 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=7826, suffix heap size=134152,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=171 2023-04-28 14:59:50,116 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=487 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/f9199e2cb98243b6937fbef15bd3c39d 2023-04-28 14:59:50,119 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/f4c341c4701a4db1a14923d04b5bd15e as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/f4c341c4701a4db1a14923d04b5bd15e 2023-04-28 14:59:50,121 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/f4c341c4701a4db1a14923d04b5bd15e, entries=150, sequenceid=487, filesize=25.4 K 2023-04-28 14:59:50,122 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/b4f2253a79684a679d741ffe1448f2c7 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/b4f2253a79684a679d741ffe1448f2c7 2023-04-28 14:59:50,126 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/b4f2253a79684a679d741ffe1448f2c7, entries=150, sequenceid=487, filesize=11.9 K 2023-04-28 14:59:50,127 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/f9199e2cb98243b6937fbef15bd3c39d as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/f9199e2cb98243b6937fbef15bd3c39d 2023-04-28 14:59:50,129 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/f9199e2cb98243b6937fbef15bd3c39d, entries=150, sequenceid=487, filesize=11.9 K 2023-04-28 14:59:50,130 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~65.34 KB/66912, currentSize=102.87 KB/105340 for 2e61d8a0a5c8be60fbc343435192fa13 in 457ms, sequenceid=487, compaction requested=true 2023-04-28 14:59:50,130 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:50,130 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:50,130 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:50,130 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:59:50,130 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:59:50,130 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:50,131 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 71636 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:59:50,131 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34660 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:59:50,131 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): 2e61d8a0a5c8be60fbc343435192fa13/A is initiating minor compaction (all files) 2023-04-28 14:59:50,131 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 2e61d8a0a5c8be60fbc343435192fa13/B is initiating minor compaction (all files) 2023-04-28 14:59:50,131 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e61d8a0a5c8be60fbc343435192fa13/A in TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:50,131 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e61d8a0a5c8be60fbc343435192fa13/B in TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:50,131 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/989d9db29d234a7bab2e46834ef13b30, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/3f019e3e5f9346319065db6dff0a2cff, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/f4c341c4701a4db1a14923d04b5bd15e] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp, totalSize=70.0 K 2023-04-28 14:59:50,131 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/7036e801e15e4a648b3347159b70de5d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/5da7fdde2e1b4e2ba8e553a09924a365, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/b4f2253a79684a679d741ffe1448f2c7] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp, totalSize=33.8 K 2023-04-28 14:59:50,131 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 989d9db29d234a7bab2e46834ef13b30, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682693986405 2023-04-28 14:59:50,131 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 7036e801e15e4a648b3347159b70de5d, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682693986405 2023-04-28 14:59:50,132 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 3f019e3e5f9346319065db6dff0a2cff, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1682693987021 2023-04-28 14:59:50,132 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 5da7fdde2e1b4e2ba8e553a09924a365, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1682693987021 2023-04-28 14:59:50,132 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting f4c341c4701a4db1a14923d04b5bd15e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1682693988553 2023-04-28 14:59:50,132 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting b4f2253a79684a679d741ffe1448f2c7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1682693988553 2023-04-28 14:59:50,136 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 2e61d8a0a5c8be60fbc343435192fa13#B#compaction#489 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:59:50,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:50,137 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2e61d8a0a5c8be60fbc343435192fa13 3/3 column families, dataSize=109.58 KB heapSize=304.56 KB 2023-04-28 14:59:50,142 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2e61d8a0a5c8be60fbc343435192fa13#A#compaction#490 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:59:50,142 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=A 2023-04-28 14:59:50,143 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:59:50,143 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=B 2023-04-28 14:59:50,143 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:59:50,143 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=C 2023-04-28 14:59:50,143 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:59:50,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:50,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39988 deadline: 1682694050151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:50,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39948 deadline: 1682694050153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:50,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:40006 deadline: 1682694050154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,166 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/866284dbaab24afd8cf713ea9d513582 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/866284dbaab24afd8cf713ea9d513582 2023-04-28 14:59:50,167 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304289ba5e6d203d14539af30404dbca9c1b1 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304289ba5e6d203d14539af30404dbca9c1b1 2023-04-28 14:59:50,170 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2e61d8a0a5c8be60fbc343435192fa13/B of 2e61d8a0a5c8be60fbc343435192fa13 into 866284dbaab24afd8cf713ea9d513582(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:50,170 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:50,170 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13., storeName=2e61d8a0a5c8be60fbc343435192fa13/B, priority=13, startTime=1682693990130; duration=0sec 2023-04-28 14:59:50,170 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:50,170 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 14:59:50,171 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34660 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 14:59:50,171 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 2e61d8a0a5c8be60fbc343435192fa13/C is initiating minor compaction (all files) 2023-04-28 14:59:50,171 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e61d8a0a5c8be60fbc343435192fa13/C in TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:50,171 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/b2a3a124263143b2b1088e65493da976, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/2b59847a41794c7590d6d027a9b0ba79, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/f9199e2cb98243b6937fbef15bd3c39d] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp, totalSize=33.8 K 2023-04-28 14:59:50,171 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting b2a3a124263143b2b1088e65493da976, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682693986405 2023-04-28 14:59:50,172 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 2b59847a41794c7590d6d027a9b0ba79, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1682693987021 2023-04-28 14:59:50,172 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting f9199e2cb98243b6937fbef15bd3c39d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1682693988553 2023-04-28 14:59:50,182 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=830, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/37d4a88beed04589b60abf79bc96942d 2023-04-28 14:59:50,183 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 2e61d8a0a5c8be60fbc343435192fa13#C#compaction#492 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 14:59:50,193 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:50,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:50,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:50,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:50,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:50,198 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:50,198 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:50,198 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:50,198 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:50,199 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,199 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:50,199 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,199 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:50,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:50,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:50,200 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,200 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:50,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:50,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:50,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:50,201 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:50,201 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/60c7a60448af445cafe3a7a2a8cf60c4 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/60c7a60448af445cafe3a7a2a8cf60c4 2023-04-28 14:59:50,202 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:50,202 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:50,202 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:50,202 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:50,202 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:50,204 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2e61d8a0a5c8be60fbc343435192fa13/C of 2e61d8a0a5c8be60fbc343435192fa13 into 60c7a60448af445cafe3a7a2a8cf60c4(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:50,205 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:50,205 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13., storeName=2e61d8a0a5c8be60fbc343435192fa13/C, priority=13, startTime=1682693990130; duration=0sec 2023-04-28 14:59:50,205 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:50,207 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:50,207 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:50,207 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:50,207 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:50,207 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:50,207 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:50,207 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:50,207 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:50,207 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,207 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:50,207 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:50,208 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:50,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,208 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:50,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:50,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:50,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:50,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21c043b0[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:50,209 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:50,209 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,210 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,210 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:50,210 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:50,210 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:50,210 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,210 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:50,211 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:50,211 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:50,211 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,211 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:50,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:50,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:50,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:50,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:50,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:50,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:50,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:50,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:50,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:50,213 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:50,213 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:50,213 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:50,213 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:50,213 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:50,214 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,214 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,214 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:50,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:50,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:50,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:50,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:50,214 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:50,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:50,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:50,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:50,216 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:50,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:50,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,217 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,217 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:50,217 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:50,217 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:50,217 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:50,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:50,217 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:50,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:50,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:50,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:50,217 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,217 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,217 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,217 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,218 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:50,218 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:50,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:50,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:50,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39988 deadline: 1682694050255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:50,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39948 deadline: 1682694050257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:50,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:40006 deadline: 1682694050258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,318 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:50,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:50,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:50,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:50,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:50,320 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:50,320 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:50,320 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:50,321 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:50,321 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,321 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:50,321 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:50,322 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:50,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:50,322 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,322 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:50,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:50,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:50,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:50,322 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:50,323 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:50,323 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:50,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:50,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:50,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:50,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:50,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:50,323 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:50,323 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:50,323 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:50,323 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:50,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:50,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:50,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:50,323 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:50,324 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,324 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:50,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:50,324 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,324 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:50,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:50,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:50,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:50,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:50,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:50,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@528226d1[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:50,325 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:50,325 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,326 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,327 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:50,327 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:50,327 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:50,327 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,327 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:50,327 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:50,327 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,327 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:50,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:50,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:50,327 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:50,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:50,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:50,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:50,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:50,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:50,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:50,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:50,330 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:50,330 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:50,330 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:50,330 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:50,330 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:50,331 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:50,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:50,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:50,332 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,332 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:50,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:50,332 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:50,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:50,334 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:50,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,335 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:50,335 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:50,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:50,335 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,336 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:50,336 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:50,336 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:50,336 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:50,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:50,336 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:50,336 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:50,336 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,336 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:50,336 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:50,436 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:50,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:50,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:50,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:50,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:50,439 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:50,439 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:50,439 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:50,439 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:50,439 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,439 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:50,440 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,440 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:50,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:50,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:50,440 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,440 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:50,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:50,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:50,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:50,441 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:50,441 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:50,441 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:50,441 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:50,441 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:50,441 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:50,441 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:50,441 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:50,441 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:50,441 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:50,441 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:50,441 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:50,441 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:50,441 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:50,442 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,442 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:50,442 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:50,442 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,442 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:50,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:50,443 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,443 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:50,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:50,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:50,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:50,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:50,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:50,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11952347[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:50,444 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:50,444 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,444 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,444 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:50,444 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:50,445 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:50,445 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,445 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:50,445 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,445 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:50,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:50,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:50,445 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:50,445 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:50,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:50,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:50,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:50,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:50,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:50,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:50,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:50,447 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:50,447 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:50,447 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:50,447 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:50,447 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:50,448 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,448 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:50,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,449 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:50,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:50,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:50,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:50,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:50,449 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:50,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:50,450 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:50,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:50,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,451 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,451 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:50,451 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:50,451 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:50,452 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:50,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:50,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:50,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:50,452 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:50,452 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:50,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:50,452 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:50,452 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,452 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:50,452 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,452 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:50,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:50,459 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:50,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39988 deadline: 1682694050457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:50,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:40006 deadline: 1682694050459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:50,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39948 deadline: 1682694050460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,552 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:50,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:50,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:50,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:50,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:50,555 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:50,555 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:50,555 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:50,555 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:50,555 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,555 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:50,556 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:50,556 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:50,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:50,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:50,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,556 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,556 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:50,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:50,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:50,557 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:50,557 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:50,557 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:50,557 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:50,557 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:50,557 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:50,557 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:50,557 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:50,557 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:50,557 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:50,557 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:50,557 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:50,557 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:50,557 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:50,557 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,557 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:50,557 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:50,558 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:50,558 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:50,559 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:50,559 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:50,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:50,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f136805[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:50,560 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:50,560 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,560 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,561 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:50,561 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:50,561 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:50,561 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:50,561 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,561 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:50,562 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:50,562 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:50,562 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:50,562 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:50,562 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:50,562 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:50,562 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:50,562 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:50,563 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,563 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,563 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:50,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:50,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:50,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:50,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:50,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:50,563 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:50,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:50,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:50,565 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:50,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:50,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,566 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:50,566 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:50,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:50,566 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:50,566 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:50,566 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:50,566 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,566 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,566 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:50,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:50,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:50,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:50,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:50,566 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:50,567 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,567 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:50,567 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:50,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:50,608 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=830 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/b46745ea67e746e3937de216cf95e50d 2023-04-28 14:59:50,616 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=830 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/722bbaea5d304bf6b37fcfeccfbfb4d7 2023-04-28 14:59:50,618 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/37d4a88beed04589b60abf79bc96942d as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/37d4a88beed04589b60abf79bc96942d 2023-04-28 14:59:50,621 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/37d4a88beed04589b60abf79bc96942d, entries=200, sequenceid=830, filesize=32.3 K 2023-04-28 14:59:50,622 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/b46745ea67e746e3937de216cf95e50d as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/b46745ea67e746e3937de216cf95e50d 2023-04-28 14:59:50,624 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/b46745ea67e746e3937de216cf95e50d, entries=150, sequenceid=830, filesize=11.9 K 2023-04-28 14:59:50,625 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/722bbaea5d304bf6b37fcfeccfbfb4d7 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/722bbaea5d304bf6b37fcfeccfbfb4d7 2023-04-28 14:59:50,628 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/722bbaea5d304bf6b37fcfeccfbfb4d7, entries=150, sequenceid=830, filesize=11.9 K 2023-04-28 14:59:50,628 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~123.00 KB/125950, heapSize ~344.36 KB/352624, currentSize=60.38 KB/61830 for 2e61d8a0a5c8be60fbc343435192fa13 in 491ms, sequenceid=830, compaction requested=false 2023-04-28 14:59:50,629 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:50,667 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:50,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:50,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:50,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:50,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:50,670 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:50,670 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:50,670 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:50,670 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:50,670 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,670 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:50,671 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:50,671 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:50,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:50,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:50,671 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,671 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:50,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:50,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:50,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:50,672 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:50,672 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:50,672 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:50,672 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:50,672 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:50,672 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:50,672 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:50,672 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:50,672 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:50,672 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:50,672 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2e61d8a0a5c8be60fbc343435192fa13 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-28 14:59:50,673 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=A 2023-04-28 14:59:50,673 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:50,673 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=B 2023-04-28 14:59:50,673 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:50,673 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=C 2023-04-28 14:59:50,673 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:50,686 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428a950d5a7766b43fea4ae438d82793fdc to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a950d5a7766b43fea4ae438d82793fdc 2023-04-28 14:59:50,690 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=844, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/a93f3a182b394bcf8379ad20ed452be2 2023-04-28 14:59:50,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:50,762 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:50,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:50,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39948 deadline: 1682694050781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693990776, max timestamp=1682693990783Num uniques 859; 2023-04-28 14:59:50,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:59:50,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693990776, max timestamp=1682693990783Num uniques 859; 2023-04-28 14:59:50,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:59:50,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682693990776, max timestamp=1682693990783Num uniques 859; 2023-04-28 14:59:50,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 14:59:50,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:50,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:40006 deadline: 1682694050808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:50,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39988 deadline: 1682694050809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,886 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:50,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39948 deadline: 1682694050884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:50,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:40006 deadline: 1682694050912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:50,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39988 deadline: 1682694050913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:50,966 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/25ba7153b330469d826c886e051af078 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/25ba7153b330469d826c886e051af078 2023-04-28 14:59:50,969 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2e61d8a0a5c8be60fbc343435192fa13/A of 2e61d8a0a5c8be60fbc343435192fa13 into 25ba7153b330469d826c886e051af078(size=26.1 K), total size for store is 58.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:50,970 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:50,970 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13., storeName=2e61d8a0a5c8be60fbc343435192fa13/A, priority=13, startTime=1682693990130; duration=0sec 2023-04-28 14:59:50,970 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:51,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:51,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39948 deadline: 1682694051087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:51,103 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=844 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/fa6f8e9d5b9f490fa8932b0a4881f72d 2023-04-28 14:59:51,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:51,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:51,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39988 deadline: 1682694051115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:51,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:40006 deadline: 1682694051115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:51,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:51,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39972 deadline: 1682694051171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:51,175 DEBUG [Thread-3274] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4132 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13., hostname=jenkins-hbase17.apache.org,37181,1682693812482, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 14:59:51,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:51,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39960 deadline: 1682694051185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:51,187 DEBUG [Thread-3280] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4146 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13., hostname=jenkins-hbase17.apache.org,37181,1682693812482, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 14:59:51,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:51,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39948 deadline: 1682694051390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:51,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:51,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39988 deadline: 1682694051419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:51,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:51,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:40006 deadline: 1682694051419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:51,510 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=844 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/5f8b64e90b9d4770a01feb66401f6dd1 2023-04-28 14:59:51,513 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/a93f3a182b394bcf8379ad20ed452be2 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/a93f3a182b394bcf8379ad20ed452be2 2023-04-28 14:59:51,515 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/a93f3a182b394bcf8379ad20ed452be2, entries=150, sequenceid=844, filesize=25.4 K 2023-04-28 14:59:51,516 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/fa6f8e9d5b9f490fa8932b0a4881f72d as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/fa6f8e9d5b9f490fa8932b0a4881f72d 2023-04-28 14:59:51,519 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/fa6f8e9d5b9f490fa8932b0a4881f72d, entries=150, sequenceid=844, filesize=11.9 K 2023-04-28 14:59:51,519 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/5f8b64e90b9d4770a01feb66401f6dd1 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/5f8b64e90b9d4770a01feb66401f6dd1 2023-04-28 14:59:51,521 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/5f8b64e90b9d4770a01feb66401f6dd1, entries=150, sequenceid=844, filesize=11.9 K 2023-04-28 14:59:51,522 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 2e61d8a0a5c8be60fbc343435192fa13 in 850ms, sequenceid=844, compaction requested=true 2023-04-28 14:59:51,522 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:51,522 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:51,522 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:51,522 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:51,522 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:51,522 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:51,522 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:51,523 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:51,523 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:51,523 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:51,523 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:51,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:51,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:51,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:51,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:51,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:51,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c6d7d94[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:51,524 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:51,524 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:51,525 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:51,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:51,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:51,525 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:51,525 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:51,525 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:51,525 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:51,525 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:51,526 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:51,526 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:51,526 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:51,526 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:51,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:51,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:51,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:51,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:51,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:51,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:51,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:51,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:51,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:51,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:51,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:51,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:51,528 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:51,528 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:51,528 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:51,528 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:51,528 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:51,529 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:51,529 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:51,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:51,529 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:51,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:51,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:51,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:51,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:51,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:51,530 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:51,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:51,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:51,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:51,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:51,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:51,531 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:51,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:51,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:51,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:51,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:51,532 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:51,532 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:51,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:51,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:51,532 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:51,532 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:51,532 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:51,532 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:51,532 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:51,532 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:51,533 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:51,533 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:51,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:51,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:51,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 14:59:51,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:51,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:51,533 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:51,533 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:51,633 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:51,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:51,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-28 14:59:51,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 14:59:51,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 14:59:51,635 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 14:59:51,635 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:51,635 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 14:59:51,635 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 14:59:51,636 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:51,636 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:51,636 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:51,636 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:51,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:51,637 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:51,637 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 14:59:51,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:51,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:51,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 14:59:51,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:51,637 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 14:59:51,637 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 14:59:51,637 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 14:59:51,637 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 14:59:51,637 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 14:59:51,637 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 14:59:51,638 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 14:59:51,638 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 14:59:51,638 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:51,638 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. started... 2023-04-28 14:59:51,638 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2e61d8a0a5c8be60fbc343435192fa13 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-04-28 14:59:51,638 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=A 2023-04-28 14:59:51,638 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:59:51,638 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=B 2023-04-28 14:59:51,638 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:59:51,638 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=C 2023-04-28 14:59:51,638 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 14:59:51,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:51,897 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. as already flushing 2023-04-28 14:59:51,922 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:51,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39948 deadline: 1682694051920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:51,922 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:51,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:40006 deadline: 1682694051922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:51,925 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:51,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37181] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39988 deadline: 1682694051924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:51,931 DEBUG [Thread-3285] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x070ab5f5 to 127.0.0.1:61762 2023-04-28 14:59:51,932 DEBUG [Thread-3285] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:59:51,932 DEBUG [Thread-3287] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2daeb795 to 127.0.0.1:61762 2023-04-28 14:59:51,932 DEBUG [Thread-3287] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:59:51,933 DEBUG [Thread-3289] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x28ac2b09 to 127.0.0.1:61762 2023-04-28 14:59:51,933 DEBUG [Thread-3289] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:59:51,936 DEBUG [Thread-3283] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d74fc48 to 127.0.0.1:61762 2023-04-28 14:59:51,936 DEBUG [Thread-3283] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:59:51,936 DEBUG [Thread-3291] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7bf1c01c to 127.0.0.1:61762 2023-04-28 14:59:51,936 DEBUG [Thread-3291] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:59:52,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:52,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39948 deadline: 1682694052023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:52,051 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428ecac8c8cab22478697bebfcd25f44bd4 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ecac8c8cab22478697bebfcd25f44bd4 2023-04-28 14:59:52,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:52,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39948 deadline: 1682694052226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:52,456 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=879, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/2ae715d4aad44a0c9a755c739986766d 2023-04-28 14:59:52,528 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:52,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39948 deadline: 1682694052528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:52,869 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=879 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/f3d3aac2a4ad46a1bb332f2602a5c4ad 2023-04-28 14:59:52,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:52,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:40006 deadline: 1682694052927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:52,928 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:52,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 342 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39988 deadline: 1682694052928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:53,030 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 14:59:53,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39948 deadline: 1682694053030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:53,278 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=879 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/fee037edfe844ac5b2a5eff8c790038e 2023-04-28 14:59:53,287 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/2ae715d4aad44a0c9a755c739986766d as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/2ae715d4aad44a0c9a755c739986766d 2023-04-28 14:59:53,293 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/2ae715d4aad44a0c9a755c739986766d, entries=150, sequenceid=879, filesize=25.4 K 2023-04-28 14:59:53,294 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/f3d3aac2a4ad46a1bb332f2602a5c4ad as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/f3d3aac2a4ad46a1bb332f2602a5c4ad 2023-04-28 14:59:53,299 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/f3d3aac2a4ad46a1bb332f2602a5c4ad, entries=150, sequenceid=879, filesize=11.9 K 2023-04-28 14:59:53,299 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/fee037edfe844ac5b2a5eff8c790038e as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/fee037edfe844ac5b2a5eff8c790038e 2023-04-28 14:59:53,304 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/fee037edfe844ac5b2a5eff8c790038e, entries=150, sequenceid=879, filesize=11.9 K 2023-04-28 14:59:53,305 INFO [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=60.38 KB/61830 for 2e61d8a0a5c8be60fbc343435192fa13 in 1667ms, sequenceid=879, compaction requested=true 2023-04-28 14:59:53,305 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:53,305 DEBUG [rs(jenkins-hbase17.apache.org,37181,1682693812482)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:53,305 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 14:59:53,305 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 14:59:53,305 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:53,305 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 14:59:53,305 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 14:59:53,307 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:53,307 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:53,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:53,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:53,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:53,307 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:53,307 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 14:59:53,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:53,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:53,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:53,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:53,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:53,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,37181,1682693812482' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 14:59:53,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f1a7866[Count = 0] remaining members to acquire global barrier 2023-04-28 14:59:53,310 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 14:59:53,310 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:53,311 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:53,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:53,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:53,311 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 14:59:53,311 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 14:59:53,311 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,37181,1682693812482' in zk 2023-04-28 14:59:53,311 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:53,311 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 14:59:53,312 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:53,312 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 14:59:53,312 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:53,313 DEBUG [member: 'jenkins-hbase17.apache.org,37181,1682693812482' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 14:59:53,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:53,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:53,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:53,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:53,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:53,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:53,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:53,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:53,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:53,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:53,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,37181,1682693812482': 2023-04-28 14:59:53,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,37181,1682693812482' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 14:59:53,317 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 14:59:53,317 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 14:59:53,317 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 14:59:53,317 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 14:59:53,317 INFO [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 14:59:53,318 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:53,318 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:53,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:53,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 14:59:53,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 14:59:53,318 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:53,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:53,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:53,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 14:59:53,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:53,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:53,319 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:53,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:53,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:53,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 14:59:53,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:53,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:53,322 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:53,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 14:59:53,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 14:59:53,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:53,325 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:53,325 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 14:59:53,325 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 14:59:53,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 14:59:53,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 14:59:53,325 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:53,325 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 14:59:53,325 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:53,325 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:53,325 DEBUG [(jenkins-hbase17.apache.org,37045,1682693811371)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 14:59:53,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 14:59:53,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 14:59:53,326 DEBUG [Thread-3282] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 14:59:53,326 DEBUG [Thread-3282] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 14:59:53,325 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 14:59:53,326 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 14:59:53,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 14:59:53,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 14:59:53,426 DEBUG [Thread-3282] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 14:59:53,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 14:59:53,684 DEBUG [regionserver/jenkins-hbase17:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.07692307692307693, tune throughput to 53.85 MB/second 2023-04-28 14:59:54,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37181] regionserver.HRegion(9158): Flush requested on 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:54,040 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2e61d8a0a5c8be60fbc343435192fa13 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-28 14:59:54,040 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=A 2023-04-28 14:59:54,040 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:54,040 DEBUG [Thread-3272] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6722b207 to 127.0.0.1:61762 2023-04-28 14:59:54,041 DEBUG [Thread-3272] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:59:54,040 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=B 2023-04-28 14:59:54,041 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:54,041 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=C 2023-04-28 14:59:54,041 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:54,462 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428b5e168cd34194a7d812ed41de4bdabde to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428b5e168cd34194a7d812ed41de4bdabde 2023-04-28 14:59:54,871 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=892, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/a20aa10f266f4829ab2d6de74bfac845 2023-04-28 14:59:54,888 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=892 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/81a65858f23f4576ae1edb714c0a1332 2023-04-28 14:59:54,895 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=892 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/21e9632ef51f48b2b751ab494d729ad6 2023-04-28 14:59:54,900 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/a20aa10f266f4829ab2d6de74bfac845 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/a20aa10f266f4829ab2d6de74bfac845 2023-04-28 14:59:54,903 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/a20aa10f266f4829ab2d6de74bfac845, entries=150, sequenceid=892, filesize=25.4 K 2023-04-28 14:59:54,903 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/81a65858f23f4576ae1edb714c0a1332 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/81a65858f23f4576ae1edb714c0a1332 2023-04-28 14:59:54,906 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/81a65858f23f4576ae1edb714c0a1332, entries=150, sequenceid=892, filesize=11.9 K 2023-04-28 14:59:54,906 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/21e9632ef51f48b2b751ab494d729ad6 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/21e9632ef51f48b2b751ab494d729ad6 2023-04-28 14:59:54,908 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/21e9632ef51f48b2b751ab494d729ad6, entries=150, sequenceid=892, filesize=11.9 K 2023-04-28 14:59:54,909 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for 2e61d8a0a5c8be60fbc343435192fa13 in 869ms, sequenceid=892, compaction requested=true 2023-04-28 14:59:54,909 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:54,909 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:54,909 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 14:59:54,909 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 14:59:54,909 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:54,909 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:54,910 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 137918 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 14:59:54,910 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61578 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 14:59:54,910 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1912): 2e61d8a0a5c8be60fbc343435192fa13/A is initiating minor compaction (all files) 2023-04-28 14:59:54,910 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 2e61d8a0a5c8be60fbc343435192fa13/B is initiating minor compaction (all files) 2023-04-28 14:59:54,910 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e61d8a0a5c8be60fbc343435192fa13/A in TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:54,910 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e61d8a0a5c8be60fbc343435192fa13/B in TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:54,910 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/25ba7153b330469d826c886e051af078, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/37d4a88beed04589b60abf79bc96942d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/a93f3a182b394bcf8379ad20ed452be2, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/2ae715d4aad44a0c9a755c739986766d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/a20aa10f266f4829ab2d6de74bfac845] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp, totalSize=134.7 K 2023-04-28 14:59:54,910 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/866284dbaab24afd8cf713ea9d513582, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/b46745ea67e746e3937de216cf95e50d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/fa6f8e9d5b9f490fa8932b0a4881f72d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/f3d3aac2a4ad46a1bb332f2602a5c4ad, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/81a65858f23f4576ae1edb714c0a1332] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp, totalSize=60.1 K 2023-04-28 14:59:54,910 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 25ba7153b330469d826c886e051af078, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1682693988553 2023-04-28 14:59:54,910 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 866284dbaab24afd8cf713ea9d513582, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1682693988553 2023-04-28 14:59:54,910 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 37d4a88beed04589b60abf79bc96942d, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=830, earliestPutTs=1682693990127 2023-04-28 14:59:54,911 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting b46745ea67e746e3937de216cf95e50d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=830, earliestPutTs=1682693990127 2023-04-28 14:59:54,911 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting a93f3a182b394bcf8379ad20ed452be2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=844, earliestPutTs=1682693990149 2023-04-28 14:59:54,911 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting fa6f8e9d5b9f490fa8932b0a4881f72d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=844, earliestPutTs=1682693990149 2023-04-28 14:59:54,911 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting 2ae715d4aad44a0c9a755c739986766d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=879, earliestPutTs=1682693990798 2023-04-28 14:59:54,911 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting f3d3aac2a4ad46a1bb332f2602a5c4ad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=879, earliestPutTs=1682693990798 2023-04-28 14:59:54,911 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] compactions.Compactor(207): Compacting a20aa10f266f4829ab2d6de74bfac845, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=892, earliestPutTs=1682693991904 2023-04-28 14:59:54,911 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 81a65858f23f4576ae1edb714c0a1332, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=892, earliestPutTs=1682693991904 2023-04-28 14:59:54,919 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 2e61d8a0a5c8be60fbc343435192fa13#B#compaction#504 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-28 14:59:54,922 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2e61d8a0a5c8be60fbc343435192fa13#A#compaction#505 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-28 14:59:54,928 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/4450c81bb27e445ea92f9ac38dea5d30 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/4450c81bb27e445ea92f9ac38dea5d30 2023-04-28 14:59:54,932 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2e61d8a0a5c8be60fbc343435192fa13/B of 2e61d8a0a5c8be60fbc343435192fa13 into 4450c81bb27e445ea92f9ac38dea5d30(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:54,932 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:54,932 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13., storeName=2e61d8a0a5c8be60fbc343435192fa13/B, priority=11, startTime=1682693994909; duration=0sec 2023-04-28 14:59:54,933 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 14:59:54,933 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 14:59:54,934 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61578 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 14:59:54,934 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1912): 2e61d8a0a5c8be60fbc343435192fa13/C is initiating minor compaction (all files) 2023-04-28 14:59:54,934 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e61d8a0a5c8be60fbc343435192fa13/C in TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:54,935 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/60c7a60448af445cafe3a7a2a8cf60c4, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/722bbaea5d304bf6b37fcfeccfbfb4d7, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/5f8b64e90b9d4770a01feb66401f6dd1, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/fee037edfe844ac5b2a5eff8c790038e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/21e9632ef51f48b2b751ab494d729ad6] into tmpdir=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp, totalSize=60.1 K 2023-04-28 14:59:54,935 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 60c7a60448af445cafe3a7a2a8cf60c4, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1682693988553 2023-04-28 14:59:54,935 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 722bbaea5d304bf6b37fcfeccfbfb4d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=830, earliestPutTs=1682693990127 2023-04-28 14:59:54,936 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 5f8b64e90b9d4770a01feb66401f6dd1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=844, earliestPutTs=1682693990149 2023-04-28 14:59:54,936 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting fee037edfe844ac5b2a5eff8c790038e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=879, earliestPutTs=1682693990798 2023-04-28 14:59:54,936 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] compactions.Compactor(207): Compacting 21e9632ef51f48b2b751ab494d729ad6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=892, earliestPutTs=1682693991904 2023-04-28 14:59:54,936 DEBUG [Thread-3276] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08d3d8bc to 127.0.0.1:61762 2023-04-28 14:59:54,937 DEBUG [Thread-3276] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:59:54,938 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/fe78ddf9904b43f29ec39067f2a2e6cf as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/fe78ddf9904b43f29ec39067f2a2e6cf 2023-04-28 14:59:54,942 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2e61d8a0a5c8be60fbc343435192fa13/A of 2e61d8a0a5c8be60fbc343435192fa13 into fe78ddf9904b43f29ec39067f2a2e6cf(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:54,942 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:54,942 INFO [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13., storeName=2e61d8a0a5c8be60fbc343435192fa13/A, priority=11, startTime=1682693994909; duration=0sec 2023-04-28 14:59:54,942 DEBUG [RS:0;jenkins-hbase17:37181-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:54,945 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] throttle.PressureAwareThroughputController(145): 2e61d8a0a5c8be60fbc343435192fa13#C#compaction#506 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-28 14:59:54,947 DEBUG [Thread-3278] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e6ad437 to 127.0.0.1:61762 2023-04-28 14:59:54,947 DEBUG [Thread-3278] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:59:55,206 DEBUG [Thread-3274] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x48502121 to 127.0.0.1:61762 2023-04-28 14:59:55,207 DEBUG [Thread-3274] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:59:55,211 DEBUG [Thread-3280] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2f6f7187 to 127.0.0.1:61762 2023-04-28 14:59:55,211 DEBUG [Thread-3280] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:59:55,211 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-28 14:59:55,211 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(394): wrote 194 2023-04-28 14:59:55,211 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(394): wrote 157 2023-04-28 14:59:55,211 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(394): wrote 150 2023-04-28 14:59:55,211 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(394): wrote 255 2023-04-28 14:59:55,211 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(394): wrote 64 2023-04-28 14:59:55,211 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-28 14:59:55,212 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(398): read 5811 2023-04-28 14:59:55,212 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(398): read 5633 2023-04-28 14:59:55,212 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(398): read 5695 2023-04-28 14:59:55,212 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(398): read 5814 2023-04-28 14:59:55,212 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(398): read 5613 2023-04-28 14:59:55,212 INFO [Listener at localhost.localdomain/40193] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-28 14:59:55,212 INFO [Listener at localhost.localdomain/40193] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-28 14:59:55,212 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50131d14 to 127.0.0.1:61762 2023-04-28 14:59:55,212 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:59:55,213 INFO [Listener at localhost.localdomain/40193] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-28 14:59:55,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.HMaster$11(2418): Client=jenkins//136.243.18.41 disable TestAcidGuarantees 2023-04-28 14:59:55,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-28 14:59:55,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-28 14:59:55,217 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682693995217"}]},"ts":"1682693995217"} 2023-04-28 14:59:55,218 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-28 14:59:55,220 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-28 14:59:55,221 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2e61d8a0a5c8be60fbc343435192fa13, UNASSIGN}] 2023-04-28 14:59:55,222 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2e61d8a0a5c8be60fbc343435192fa13, UNASSIGN 2023-04-28 14:59:55,222 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=2e61d8a0a5c8be60fbc343435192fa13, regionState=CLOSING, regionLocation=jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:55,223 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693995222"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682693995222"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682693995222"}]},"ts":"1682693995222"} 2023-04-28 14:59:55,224 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482}] 2023-04-28 14:59:55,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-28 14:59:55,364 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/78518125db3b427bb0ccecc3ee939b4f as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/78518125db3b427bb0ccecc3ee939b4f 2023-04-28 14:59:55,372 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2e61d8a0a5c8be60fbc343435192fa13/C of 2e61d8a0a5c8be60fbc343435192fa13 into 78518125db3b427bb0ccecc3ee939b4f(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 14:59:55,372 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:55,372 INFO [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13., storeName=2e61d8a0a5c8be60fbc343435192fa13/C, priority=11, startTime=1682693994909; duration=0sec 2023-04-28 14:59:55,372 DEBUG [RS:0;jenkins-hbase17:37181-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 14:59:55,376 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(109): Close 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:55,376 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1604): Closing 2e61d8a0a5c8be60fbc343435192fa13, disabling compactions & flushes 2023-04-28 14:59:55,376 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:55,376 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:55,377 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. after waiting 0 ms 2023-04-28 14:59:55,377 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:55,377 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2745): Flushing 2e61d8a0a5c8be60fbc343435192fa13 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-28 14:59:55,377 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=A 2023-04-28 14:59:55,377 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:55,377 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=B 2023-04-28 14:59:55,377 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:55,377 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e61d8a0a5c8be60fbc343435192fa13, store=C 2023-04-28 14:59:55,377 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 14:59:55,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-28 14:59:55,796 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042820139a1d6c2942588bc0001c24a0b2b6 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042820139a1d6c2942588bc0001c24a0b2b6 2023-04-28 14:59:55,806 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=902, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/63e54ea4cc424403b1ca96f01fa8ff4a 2023-04-28 14:59:55,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-28 14:59:56,219 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=902 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/7c75952783634bf0a9a5cbed92a93842 2023-04-28 14:59:56,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-28 14:59:56,636 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=902 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/9682ed588c964cdba530fcfa6ef0345e 2023-04-28 14:59:56,641 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/A/63e54ea4cc424403b1ca96f01fa8ff4a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/63e54ea4cc424403b1ca96f01fa8ff4a 2023-04-28 14:59:56,645 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/63e54ea4cc424403b1ca96f01fa8ff4a, entries=100, sequenceid=902, filesize=18.6 K 2023-04-28 14:59:56,646 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/B/7c75952783634bf0a9a5cbed92a93842 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/7c75952783634bf0a9a5cbed92a93842 2023-04-28 14:59:56,651 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/7c75952783634bf0a9a5cbed92a93842, entries=100, sequenceid=902, filesize=9.5 K 2023-04-28 14:59:56,652 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/.tmp/C/9682ed588c964cdba530fcfa6ef0345e as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/9682ed588c964cdba530fcfa6ef0345e 2023-04-28 14:59:56,656 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/9682ed588c964cdba530fcfa6ef0345e, entries=100, sequenceid=902, filesize=9.5 K 2023-04-28 14:59:56,657 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 2e61d8a0a5c8be60fbc343435192fa13 in 1280ms, sequenceid=902, compaction requested=false 2023-04-28 14:59:56,682 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/d33570364e1f44098f94b6de805f1685, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/1cfb20a90be9401da2b6b159372f44bd, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/135518df887a4af1a615d8456c5f95fc, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/4a9a539bd77140bb83b19bccfcab251f, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/d8502a96e7194a66a630f54ea8bde46a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/6654e74727b64370acbfbb6acff70ff2, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/e4e17757ea25428da01bdf480c3bc995, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/debe331f8751464880416185d9c6c48d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/05b578cedafa4f9ea9f2cb403afbb973, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/31fde17a97e342d9808ffb07799c6dd1, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/4502c11ca9ca454aae4feeb6c47eefdf, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/66a8815a885c4c66bda12bdae23af7f3, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/b57a90b36764476b8fab1c53ce6e4f1e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/1aa21840d99644f78c2df38663cf9871, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/ca95429110584e329a5ea3f613fe6c15, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/3591ca20f8374d62993aa4b1c08e5b9a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/989d9db29d234a7bab2e46834ef13b30, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/3f019e3e5f9346319065db6dff0a2cff, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/25ba7153b330469d826c886e051af078, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/f4c341c4701a4db1a14923d04b5bd15e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/37d4a88beed04589b60abf79bc96942d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/a93f3a182b394bcf8379ad20ed452be2, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/2ae715d4aad44a0c9a755c739986766d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/a20aa10f266f4829ab2d6de74bfac845] to archive 2023-04-28 14:59:56,683 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 14:59:56,684 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/d33570364e1f44098f94b6de805f1685 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/d33570364e1f44098f94b6de805f1685 2023-04-28 14:59:56,685 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/1cfb20a90be9401da2b6b159372f44bd to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/1cfb20a90be9401da2b6b159372f44bd 2023-04-28 14:59:56,686 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/135518df887a4af1a615d8456c5f95fc to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/135518df887a4af1a615d8456c5f95fc 2023-04-28 14:59:56,687 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/4a9a539bd77140bb83b19bccfcab251f to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/4a9a539bd77140bb83b19bccfcab251f 2023-04-28 14:59:56,689 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/d8502a96e7194a66a630f54ea8bde46a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/d8502a96e7194a66a630f54ea8bde46a 2023-04-28 14:59:56,690 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/6654e74727b64370acbfbb6acff70ff2 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/6654e74727b64370acbfbb6acff70ff2 2023-04-28 14:59:56,691 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/e4e17757ea25428da01bdf480c3bc995 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/e4e17757ea25428da01bdf480c3bc995 2023-04-28 14:59:56,692 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/debe331f8751464880416185d9c6c48d to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/debe331f8751464880416185d9c6c48d 2023-04-28 14:59:56,693 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/05b578cedafa4f9ea9f2cb403afbb973 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/05b578cedafa4f9ea9f2cb403afbb973 2023-04-28 14:59:56,694 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/31fde17a97e342d9808ffb07799c6dd1 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/31fde17a97e342d9808ffb07799c6dd1 2023-04-28 14:59:56,695 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/4502c11ca9ca454aae4feeb6c47eefdf to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/4502c11ca9ca454aae4feeb6c47eefdf 2023-04-28 14:59:56,696 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/66a8815a885c4c66bda12bdae23af7f3 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/66a8815a885c4c66bda12bdae23af7f3 2023-04-28 14:59:56,697 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/b57a90b36764476b8fab1c53ce6e4f1e to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/b57a90b36764476b8fab1c53ce6e4f1e 2023-04-28 14:59:56,698 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/1aa21840d99644f78c2df38663cf9871 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/1aa21840d99644f78c2df38663cf9871 2023-04-28 14:59:56,700 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/ca95429110584e329a5ea3f613fe6c15 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/ca95429110584e329a5ea3f613fe6c15 2023-04-28 14:59:56,701 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/3591ca20f8374d62993aa4b1c08e5b9a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/3591ca20f8374d62993aa4b1c08e5b9a 2023-04-28 14:59:56,702 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/989d9db29d234a7bab2e46834ef13b30 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/989d9db29d234a7bab2e46834ef13b30 2023-04-28 14:59:56,703 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/3f019e3e5f9346319065db6dff0a2cff to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/3f019e3e5f9346319065db6dff0a2cff 2023-04-28 14:59:56,705 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/25ba7153b330469d826c886e051af078 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/25ba7153b330469d826c886e051af078 2023-04-28 14:59:56,706 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/f4c341c4701a4db1a14923d04b5bd15e to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/f4c341c4701a4db1a14923d04b5bd15e 2023-04-28 14:59:56,707 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/37d4a88beed04589b60abf79bc96942d to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/37d4a88beed04589b60abf79bc96942d 2023-04-28 14:59:56,709 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/a93f3a182b394bcf8379ad20ed452be2 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/a93f3a182b394bcf8379ad20ed452be2 2023-04-28 14:59:56,710 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/2ae715d4aad44a0c9a755c739986766d to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/2ae715d4aad44a0c9a755c739986766d 2023-04-28 14:59:56,711 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/a20aa10f266f4829ab2d6de74bfac845 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/a20aa10f266f4829ab2d6de74bfac845 2023-04-28 14:59:56,715 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/bc826cce2a9148969757e78d4333ace2, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/0448540e08dc45daba93ccb7c0bd3177, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/edd253c343ee48c7b483a50cd7dfb105, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/ff4a96be2f234f15ab0158baf42d012a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/f84ef14288784e078f74dc17b080441a, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/3ac1cf8e19914527af60193d0bf6d440, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/2bb0928c74a540629be04b27b9485bfe, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/34fc5b066dda4dbca2f6e47112fcb0d8, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/ef20816bc979480086a069ace82c9199, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/67658770831c4e13bb4cfd50728fe2db, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/1573848cf35346b582c90a15c494bbbd, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/3b56715e0c0946c2a9f819be389a6e90, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/39d4e5de7c724503b34bc1ee73cb4105, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/e771e56486eb4774a68cbb0c285fa615, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/acb94e7291484c02a6f0110e7b59da49, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/7036e801e15e4a648b3347159b70de5d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/898142dcaab04e3dbf1fd9c2d6ee2f23, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/5da7fdde2e1b4e2ba8e553a09924a365, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/866284dbaab24afd8cf713ea9d513582, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/b4f2253a79684a679d741ffe1448f2c7, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/b46745ea67e746e3937de216cf95e50d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/fa6f8e9d5b9f490fa8932b0a4881f72d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/f3d3aac2a4ad46a1bb332f2602a5c4ad, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/81a65858f23f4576ae1edb714c0a1332] to archive 2023-04-28 14:59:56,715 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 14:59:56,717 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/bc826cce2a9148969757e78d4333ace2 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/bc826cce2a9148969757e78d4333ace2 2023-04-28 14:59:56,718 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/0448540e08dc45daba93ccb7c0bd3177 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/0448540e08dc45daba93ccb7c0bd3177 2023-04-28 14:59:56,720 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/edd253c343ee48c7b483a50cd7dfb105 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/edd253c343ee48c7b483a50cd7dfb105 2023-04-28 14:59:56,721 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/ff4a96be2f234f15ab0158baf42d012a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/ff4a96be2f234f15ab0158baf42d012a 2023-04-28 14:59:56,722 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/f84ef14288784e078f74dc17b080441a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/f84ef14288784e078f74dc17b080441a 2023-04-28 14:59:56,724 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/3ac1cf8e19914527af60193d0bf6d440 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/3ac1cf8e19914527af60193d0bf6d440 2023-04-28 14:59:56,725 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/2bb0928c74a540629be04b27b9485bfe to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/2bb0928c74a540629be04b27b9485bfe 2023-04-28 14:59:56,726 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/34fc5b066dda4dbca2f6e47112fcb0d8 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/34fc5b066dda4dbca2f6e47112fcb0d8 2023-04-28 14:59:56,727 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/ef20816bc979480086a069ace82c9199 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/ef20816bc979480086a069ace82c9199 2023-04-28 14:59:56,728 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/67658770831c4e13bb4cfd50728fe2db to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/67658770831c4e13bb4cfd50728fe2db 2023-04-28 14:59:56,728 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/1573848cf35346b582c90a15c494bbbd to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/1573848cf35346b582c90a15c494bbbd 2023-04-28 14:59:56,729 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/3b56715e0c0946c2a9f819be389a6e90 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/3b56715e0c0946c2a9f819be389a6e90 2023-04-28 14:59:56,730 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/39d4e5de7c724503b34bc1ee73cb4105 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/39d4e5de7c724503b34bc1ee73cb4105 2023-04-28 14:59:56,731 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/e771e56486eb4774a68cbb0c285fa615 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/e771e56486eb4774a68cbb0c285fa615 2023-04-28 14:59:56,732 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/acb94e7291484c02a6f0110e7b59da49 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/acb94e7291484c02a6f0110e7b59da49 2023-04-28 14:59:56,733 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/7036e801e15e4a648b3347159b70de5d to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/7036e801e15e4a648b3347159b70de5d 2023-04-28 14:59:56,733 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/898142dcaab04e3dbf1fd9c2d6ee2f23 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/898142dcaab04e3dbf1fd9c2d6ee2f23 2023-04-28 14:59:56,734 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/5da7fdde2e1b4e2ba8e553a09924a365 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/5da7fdde2e1b4e2ba8e553a09924a365 2023-04-28 14:59:56,735 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/866284dbaab24afd8cf713ea9d513582 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/866284dbaab24afd8cf713ea9d513582 2023-04-28 14:59:56,736 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/b4f2253a79684a679d741ffe1448f2c7 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/b4f2253a79684a679d741ffe1448f2c7 2023-04-28 14:59:56,736 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/b46745ea67e746e3937de216cf95e50d to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/b46745ea67e746e3937de216cf95e50d 2023-04-28 14:59:56,737 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/fa6f8e9d5b9f490fa8932b0a4881f72d to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/fa6f8e9d5b9f490fa8932b0a4881f72d 2023-04-28 14:59:56,738 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/f3d3aac2a4ad46a1bb332f2602a5c4ad to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/f3d3aac2a4ad46a1bb332f2602a5c4ad 2023-04-28 14:59:56,739 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/81a65858f23f4576ae1edb714c0a1332 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/81a65858f23f4576ae1edb714c0a1332 2023-04-28 14:59:56,741 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/a2f36f9f09344f638c948bfe5ca77a1f, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/0d4c0e9b0f614b08a35a3e238e6ce331, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/6530f5243efe43e8ab7b410c8d2cfe99, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/3fbc2dc6e9444cf0805caa67dee50bd2, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/4b3d78faf65a46f0bd3292ca26b43c31, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/a7ff446eefe4498485e64b65be77168e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/582378578f944e6c846dc37aace5633d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/2085ae8ba5924ea886d7a21c836eb03e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/5392f5a04aab413db26535f541b26e2d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/8a61fb65bd0846dd81d2662415e0f4ec, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/9f025ae910e349f99b4b92dc145eb726, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/32961c0039564e95a60d74a55b789faf, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/19d3903afe35457097f69e4c326db980, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/1e7f4d07443f443d887cdf5ccd40844b, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/87899a1935194a4c9c24609222efcd15, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/b2a3a124263143b2b1088e65493da976, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/1f7ef88a5cc44c218bab8728e48d1244, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/2b59847a41794c7590d6d027a9b0ba79, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/60c7a60448af445cafe3a7a2a8cf60c4, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/f9199e2cb98243b6937fbef15bd3c39d, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/722bbaea5d304bf6b37fcfeccfbfb4d7, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/5f8b64e90b9d4770a01feb66401f6dd1, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/fee037edfe844ac5b2a5eff8c790038e, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/21e9632ef51f48b2b751ab494d729ad6] to archive 2023-04-28 14:59:56,742 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 14:59:56,743 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/a2f36f9f09344f638c948bfe5ca77a1f to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/a2f36f9f09344f638c948bfe5ca77a1f 2023-04-28 14:59:56,744 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/0d4c0e9b0f614b08a35a3e238e6ce331 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/0d4c0e9b0f614b08a35a3e238e6ce331 2023-04-28 14:59:56,745 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/6530f5243efe43e8ab7b410c8d2cfe99 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/6530f5243efe43e8ab7b410c8d2cfe99 2023-04-28 14:59:56,746 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/3fbc2dc6e9444cf0805caa67dee50bd2 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/3fbc2dc6e9444cf0805caa67dee50bd2 2023-04-28 14:59:56,747 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/4b3d78faf65a46f0bd3292ca26b43c31 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/4b3d78faf65a46f0bd3292ca26b43c31 2023-04-28 14:59:56,747 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/a7ff446eefe4498485e64b65be77168e to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/a7ff446eefe4498485e64b65be77168e 2023-04-28 14:59:56,748 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/582378578f944e6c846dc37aace5633d to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/582378578f944e6c846dc37aace5633d 2023-04-28 14:59:56,749 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/2085ae8ba5924ea886d7a21c836eb03e to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/2085ae8ba5924ea886d7a21c836eb03e 2023-04-28 14:59:56,749 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/5392f5a04aab413db26535f541b26e2d to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/5392f5a04aab413db26535f541b26e2d 2023-04-28 14:59:56,750 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/8a61fb65bd0846dd81d2662415e0f4ec to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/8a61fb65bd0846dd81d2662415e0f4ec 2023-04-28 14:59:56,750 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/9f025ae910e349f99b4b92dc145eb726 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/9f025ae910e349f99b4b92dc145eb726 2023-04-28 14:59:56,751 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/32961c0039564e95a60d74a55b789faf to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/32961c0039564e95a60d74a55b789faf 2023-04-28 14:59:56,752 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/19d3903afe35457097f69e4c326db980 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/19d3903afe35457097f69e4c326db980 2023-04-28 14:59:56,752 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/1e7f4d07443f443d887cdf5ccd40844b to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/1e7f4d07443f443d887cdf5ccd40844b 2023-04-28 14:59:56,753 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/87899a1935194a4c9c24609222efcd15 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/87899a1935194a4c9c24609222efcd15 2023-04-28 14:59:56,754 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/b2a3a124263143b2b1088e65493da976 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/b2a3a124263143b2b1088e65493da976 2023-04-28 14:59:56,755 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/1f7ef88a5cc44c218bab8728e48d1244 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/1f7ef88a5cc44c218bab8728e48d1244 2023-04-28 14:59:56,755 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/2b59847a41794c7590d6d027a9b0ba79 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/2b59847a41794c7590d6d027a9b0ba79 2023-04-28 14:59:56,756 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/60c7a60448af445cafe3a7a2a8cf60c4 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/60c7a60448af445cafe3a7a2a8cf60c4 2023-04-28 14:59:56,757 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/f9199e2cb98243b6937fbef15bd3c39d to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/f9199e2cb98243b6937fbef15bd3c39d 2023-04-28 14:59:56,757 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/722bbaea5d304bf6b37fcfeccfbfb4d7 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/722bbaea5d304bf6b37fcfeccfbfb4d7 2023-04-28 14:59:56,758 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/5f8b64e90b9d4770a01feb66401f6dd1 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/5f8b64e90b9d4770a01feb66401f6dd1 2023-04-28 14:59:56,759 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/fee037edfe844ac5b2a5eff8c790038e to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/fee037edfe844ac5b2a5eff8c790038e 2023-04-28 14:59:56,760 DEBUG [StoreCloser-TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/21e9632ef51f48b2b751ab494d729ad6 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/21e9632ef51f48b2b751ab494d729ad6 2023-04-28 14:59:56,764 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/recovered.edits/905.seqid, newMaxSeqId=905, maxSeqId=4 2023-04-28 14:59:56,765 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13. 2023-04-28 14:59:56,765 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1558): Region close journal for 2e61d8a0a5c8be60fbc343435192fa13: 2023-04-28 14:59:56,767 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(147): Closed 2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:56,767 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=2e61d8a0a5c8be60fbc343435192fa13, regionState=CLOSED 2023-04-28 14:59:56,767 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682693996767"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682693996767"}]},"ts":"1682693996767"} 2023-04-28 14:59:56,769 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-28 14:59:56,769 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 2e61d8a0a5c8be60fbc343435192fa13, server=jenkins-hbase17.apache.org,37181,1682693812482 in 1.5440 sec 2023-04-28 14:59:56,770 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-28 14:59:56,771 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2e61d8a0a5c8be60fbc343435192fa13, UNASSIGN in 1.5480 sec 2023-04-28 14:59:56,771 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682693996771"}]},"ts":"1682693996771"} 2023-04-28 14:59:56,772 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-28 14:59:56,774 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-28 14:59:56,775 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5610 sec 2023-04-28 14:59:57,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-28 14:59:57,325 INFO [Listener at localhost.localdomain/40193] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-28 14:59:57,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.HMaster$5(2228): Client=jenkins//136.243.18.41 delete TestAcidGuarantees 2023-04-28 14:59:57,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:59:57,328 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:59:57,328 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:59:57,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-28 14:59:57,332 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:57,334 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A, FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B, FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C, FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/recovered.edits] 2023-04-28 14:59:57,336 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/63e54ea4cc424403b1ca96f01fa8ff4a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/63e54ea4cc424403b1ca96f01fa8ff4a 2023-04-28 14:59:57,337 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/fe78ddf9904b43f29ec39067f2a2e6cf to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/A/fe78ddf9904b43f29ec39067f2a2e6cf 2023-04-28 14:59:57,339 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/4450c81bb27e445ea92f9ac38dea5d30 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/4450c81bb27e445ea92f9ac38dea5d30 2023-04-28 14:59:57,340 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/7c75952783634bf0a9a5cbed92a93842 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/B/7c75952783634bf0a9a5cbed92a93842 2023-04-28 14:59:57,342 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/78518125db3b427bb0ccecc3ee939b4f to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/78518125db3b427bb0ccecc3ee939b4f 2023-04-28 14:59:57,342 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/9682ed588c964cdba530fcfa6ef0345e to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/C/9682ed588c964cdba530fcfa6ef0345e 2023-04-28 14:59:57,344 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/recovered.edits/905.seqid to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13/recovered.edits/905.seqid 2023-04-28 14:59:57,345 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/.tmp/data/default/TestAcidGuarantees/2e61d8a0a5c8be60fbc343435192fa13 2023-04-28 14:59:57,345 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 14:59:57,345 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-28 14:59:57,346 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-28 14:59:57,348 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304281411ae3fcd0d4d04927784b0ec99c09b to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304281411ae3fcd0d4d04927784b0ec99c09b 2023-04-28 14:59:57,349 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042816cfb0a12a94495ca05b1960e08f6fd5 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042816cfb0a12a94495ca05b1960e08f6fd5 2023-04-28 14:59:57,349 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304281ba1283d4fb64d38a9feedc378c235ef to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304281ba1283d4fb64d38a9feedc378c235ef 2023-04-28 14:59:57,350 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042820139a1d6c2942588bc0001c24a0b2b6 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042820139a1d6c2942588bc0001c24a0b2b6 2023-04-28 14:59:57,351 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304282f9816b714324c1ba7bfa52e115bebc5 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304282f9816b714324c1ba7bfa52e115bebc5 2023-04-28 14:59:57,351 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042834c84c8b4dbc4c4c9a314431210a86b7 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042834c84c8b4dbc4c4c9a314431210a86b7 2023-04-28 14:59:57,352 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304284211cdd39a8f432887dee0b4f2bf61ce to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304284211cdd39a8f432887dee0b4f2bf61ce 2023-04-28 14:59:57,353 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042866520942fab24a1098ee91e3ee243916 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042866520942fab24a1098ee91e3ee243916 2023-04-28 14:59:57,353 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304287c38d8b2ace547c8ba449259ef7ecbf3 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304287c38d8b2ace547c8ba449259ef7ecbf3 2023-04-28 14:59:57,354 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304289ba5e6d203d14539af30404dbca9c1b1 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304289ba5e6d203d14539af30404dbca9c1b1 2023-04-28 14:59:57,355 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a380e730c8104b13a99c53d1daae570d to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a380e730c8104b13a99c53d1daae570d 2023-04-28 14:59:57,356 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a7ae4976a7b843558b8d974c7a23149b to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a7ae4976a7b843558b8d974c7a23149b 2023-04-28 14:59:57,356 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a950d5a7766b43fea4ae438d82793fdc to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a950d5a7766b43fea4ae438d82793fdc 2023-04-28 14:59:57,357 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428b5e168cd34194a7d812ed41de4bdabde to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428b5e168cd34194a7d812ed41de4bdabde 2023-04-28 14:59:57,358 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428c3c65d94e5e9492ea16e858855f9eaee to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428c3c65d94e5e9492ea16e858855f9eaee 2023-04-28 14:59:57,358 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428cd529552c6d94db8984c5b1b5e2fd162 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428cd529552c6d94db8984c5b1b5e2fd162 2023-04-28 14:59:57,359 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428d82014bd92244608bdcac8ee884f9746 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428d82014bd92244608bdcac8ee884f9746 2023-04-28 14:59:57,360 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428dac7446b35354c608b20fa7cc1b7366a to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428dac7446b35354c608b20fa7cc1b7366a 2023-04-28 14:59:57,360 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ecac8c8cab22478697bebfcd25f44bd4 to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ecac8c8cab22478697bebfcd25f44bd4 2023-04-28 14:59:57,361 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428fb56ccae7a6445c799f43c85957f322c to hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428fb56ccae7a6445c799f43c85957f322c 2023-04-28 14:59:57,361 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-28 14:59:57,362 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:59:57,363 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-28 14:59:57,365 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-28 14:59:57,365 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:59:57,365 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-28 14:59:57,365 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682693997365"}]},"ts":"9223372036854775807"} 2023-04-28 14:59:57,367 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-28 14:59:57,367 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 2e61d8a0a5c8be60fbc343435192fa13, NAME => 'TestAcidGuarantees,,1682693968821.2e61d8a0a5c8be60fbc343435192fa13.', STARTKEY => '', ENDKEY => ''}] 2023-04-28 14:59:57,367 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-28 14:59:57,367 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682693997367"}]},"ts":"9223372036854775807"} 2023-04-28 14:59:57,368 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-28 14:59:57,369 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 14:59:57,370 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 44 msec 2023-04-28 14:59:57,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37045] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-28 14:59:57,431 INFO [Listener at localhost.localdomain/40193] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-28 14:59:57,447 INFO [Listener at localhost.localdomain/40193] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=229 (was 228) - Thread LEAK? -, OpenFileDescriptor=548 (was 541) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=317 (was 320), ProcessCount=168 (was 168), AvailableMemoryMB=5033 (was 5006) - AvailableMemoryMB LEAK? - 2023-04-28 14:59:57,448 INFO [Listener at localhost.localdomain/40193] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-28 14:59:57,448 INFO [Listener at localhost.localdomain/40193] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-28 14:59:57,448 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7ddc537c to 127.0.0.1:61762 2023-04-28 14:59:57,448 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:59:57,448 DEBUG [Listener at localhost.localdomain/40193] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-28 14:59:57,449 DEBUG [Listener at localhost.localdomain/40193] util.JVMClusterUtil(257): Found active master hash=2135873419, stopped=false 2023-04-28 14:59:57,449 INFO [Listener at localhost.localdomain/40193] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase17.apache.org,37045,1682693811371 2023-04-28 14:59:57,449 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-28 14:59:57,449 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-28 14:59:57,449 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-28 14:59:57,449 INFO [Listener at localhost.localdomain/40193] procedure2.ProcedureExecutor(629): Stopping 2023-04-28 14:59:57,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-28 14:59:57,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-28 14:59:57,450 DEBUG [Listener at localhost.localdomain/40193] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x20638564 to 127.0.0.1:61762 2023-04-28 14:59:57,450 DEBUG [Listener at localhost.localdomain/40193] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:59:57,450 INFO [Listener at localhost.localdomain/40193] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase17.apache.org,37181,1682693812482' ***** 2023-04-28 14:59:57,450 INFO [Listener at localhost.localdomain/40193] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-28 14:59:57,451 INFO [RS:0;jenkins-hbase17:37181] regionserver.HeapMemoryManager(220): Stopping 2023-04-28 14:59:57,451 INFO [RS:0;jenkins-hbase17:37181] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-28 14:59:57,451 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-28 14:59:57,451 INFO [RS:0;jenkins-hbase17:37181] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-28 14:59:57,451 INFO [RS:0;jenkins-hbase17:37181] regionserver.HRegionServer(3304): Received CLOSE for afac81c8140d02aaf48051ad94c432ec 2023-04-28 14:59:57,451 INFO [RS:0;jenkins-hbase17:37181] regionserver.HRegionServer(1145): stopping server jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:57,451 DEBUG [RS:0;jenkins-hbase17:37181] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5b182454 to 127.0.0.1:61762 2023-04-28 14:59:57,452 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1604): Closing afac81c8140d02aaf48051ad94c432ec, disabling compactions & flushes 2023-04-28 14:59:57,452 DEBUG [RS:0;jenkins-hbase17:37181] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:59:57,452 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682693814147.afac81c8140d02aaf48051ad94c432ec. 2023-04-28 14:59:57,452 INFO [RS:0;jenkins-hbase17:37181] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-28 14:59:57,452 INFO [RS:0;jenkins-hbase17:37181] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-28 14:59:57,452 INFO [RS:0;jenkins-hbase17:37181] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-28 14:59:57,452 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682693814147.afac81c8140d02aaf48051ad94c432ec. 2023-04-28 14:59:57,452 INFO [RS:0;jenkins-hbase17:37181] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-28 14:59:57,452 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682693814147.afac81c8140d02aaf48051ad94c432ec. after waiting 0 ms 2023-04-28 14:59:57,452 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682693814147.afac81c8140d02aaf48051ad94c432ec. 2023-04-28 14:59:57,452 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2745): Flushing afac81c8140d02aaf48051ad94c432ec 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-28 14:59:57,453 INFO [RS:0;jenkins-hbase17:37181] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-28 14:59:57,453 DEBUG [RS:0;jenkins-hbase17:37181] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, afac81c8140d02aaf48051ad94c432ec=hbase:namespace,,1682693814147.afac81c8140d02aaf48051ad94c432ec.} 2023-04-28 14:59:57,453 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-28 14:59:57,453 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-28 14:59:57,453 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-28 14:59:57,453 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-28 14:59:57,453 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-28 14:59:57,453 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.94 KB heapSize=36.31 KB 2023-04-28 14:59:57,454 DEBUG [RS:0;jenkins-hbase17:37181] regionserver.HRegionServer(1505): Waiting on 1588230740, afac81c8140d02aaf48051ad94c432ec 2023-04-28 14:59:57,460 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.29 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/meta/1588230740/.tmp/info/0b5090d9329d48fc9a280ee43eaa6a9a 2023-04-28 14:59:57,464 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0b5090d9329d48fc9a280ee43eaa6a9a 2023-04-28 14:59:57,535 INFO [regionserver/jenkins-hbase17:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-28 14:59:57,655 DEBUG [RS:0;jenkins-hbase17:37181] regionserver.HRegionServer(1505): Waiting on 1588230740, afac81c8140d02aaf48051ad94c432ec 2023-04-28 14:59:57,675 INFO [regionserver/jenkins-hbase17:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-28 14:59:57,676 INFO [regionserver/jenkins-hbase17:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-28 14:59:57,855 DEBUG [RS:0;jenkins-hbase17:37181] regionserver.HRegionServer(1505): Waiting on 1588230740, afac81c8140d02aaf48051ad94c432ec 2023-04-28 14:59:57,860 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/namespace/afac81c8140d02aaf48051ad94c432ec/.tmp/info/4776381a92dd4ab49a2c580620fa4c7f 2023-04-28 14:59:57,866 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/namespace/afac81c8140d02aaf48051ad94c432ec/.tmp/info/4776381a92dd4ab49a2c580620fa4c7f as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/namespace/afac81c8140d02aaf48051ad94c432ec/info/4776381a92dd4ab49a2c580620fa4c7f 2023-04-28 14:59:57,870 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/meta/1588230740/.tmp/rep_barrier/b1beb21a3c0f4182afba69e265768b08 2023-04-28 14:59:57,871 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/namespace/afac81c8140d02aaf48051ad94c432ec/info/4776381a92dd4ab49a2c580620fa4c7f, entries=2, sequenceid=6, filesize=4.8 K 2023-04-28 14:59:57,872 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for afac81c8140d02aaf48051ad94c432ec in 420ms, sequenceid=6, compaction requested=false 2023-04-28 14:59:57,878 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b1beb21a3c0f4182afba69e265768b08 2023-04-28 14:59:57,879 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/namespace/afac81c8140d02aaf48051ad94c432ec/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-28 14:59:57,881 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682693814147.afac81c8140d02aaf48051ad94c432ec. 2023-04-28 14:59:57,881 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1558): Region close journal for afac81c8140d02aaf48051ad94c432ec: 2023-04-28 14:59:57,881 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1682693814147.afac81c8140d02aaf48051ad94c432ec. 2023-04-28 14:59:58,055 DEBUG [RS:0;jenkins-hbase17:37181] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-28 14:59:58,256 DEBUG [RS:0;jenkins-hbase17:37181] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-28 14:59:58,290 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/meta/1588230740/.tmp/table/d30468c974214bd5a2aa341083a76439 2023-04-28 14:59:58,298 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d30468c974214bd5a2aa341083a76439 2023-04-28 14:59:58,300 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/meta/1588230740/.tmp/info/0b5090d9329d48fc9a280ee43eaa6a9a as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/meta/1588230740/info/0b5090d9329d48fc9a280ee43eaa6a9a 2023-04-28 14:59:58,306 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0b5090d9329d48fc9a280ee43eaa6a9a 2023-04-28 14:59:58,306 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/meta/1588230740/info/0b5090d9329d48fc9a280ee43eaa6a9a, entries=22, sequenceid=93, filesize=7.1 K 2023-04-28 14:59:58,308 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/meta/1588230740/.tmp/rep_barrier/b1beb21a3c0f4182afba69e265768b08 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/meta/1588230740/rep_barrier/b1beb21a3c0f4182afba69e265768b08 2023-04-28 14:59:58,314 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b1beb21a3c0f4182afba69e265768b08 2023-04-28 14:59:58,314 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/meta/1588230740/rep_barrier/b1beb21a3c0f4182afba69e265768b08, entries=6, sequenceid=93, filesize=5.5 K 2023-04-28 14:59:58,316 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/meta/1588230740/.tmp/table/d30468c974214bd5a2aa341083a76439 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/meta/1588230740/table/d30468c974214bd5a2aa341083a76439 2023-04-28 14:59:58,320 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d30468c974214bd5a2aa341083a76439 2023-04-28 14:59:58,320 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/meta/1588230740/table/d30468c974214bd5a2aa341083a76439, entries=9, sequenceid=93, filesize=5.5 K 2023-04-28 14:59:58,321 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.94 KB/21446, heapSize ~36.27 KB/37136, currentSize=0 B/0 for 1588230740 in 868ms, sequenceid=93, compaction requested=false 2023-04-28 14:59:58,330 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-28 14:59:58,331 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-28 14:59:58,332 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-28 14:59:58,333 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-28 14:59:58,333 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-28 14:59:58,456 INFO [RS:0;jenkins-hbase17:37181] regionserver.HRegionServer(1171): stopping server jenkins-hbase17.apache.org,37181,1682693812482; all regions closed. 2023-04-28 14:59:58,470 DEBUG [RS:0;jenkins-hbase17:37181] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/oldWALs 2023-04-28 14:59:58,471 INFO [RS:0;jenkins-hbase17:37181] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase17.apache.org%2C37181%2C1682693812482.meta:.meta(num 1682693813924) 2023-04-28 14:59:58,475 DEBUG [RS:0;jenkins-hbase17:37181] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/oldWALs 2023-04-28 14:59:58,476 INFO [RS:0;jenkins-hbase17:37181] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase17.apache.org%2C37181%2C1682693812482:(num 1682693813822) 2023-04-28 14:59:58,476 DEBUG [RS:0;jenkins-hbase17:37181] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:59:58,476 INFO [RS:0;jenkins-hbase17:37181] regionserver.LeaseManager(133): Closed leases 2023-04-28 14:59:58,476 INFO [RS:0;jenkins-hbase17:37181] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase17:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-04-28 14:59:58,476 INFO [regionserver/jenkins-hbase17:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-28 14:59:58,477 INFO [RS:0;jenkins-hbase17:37181] ipc.NettyRpcServer(158): Stopping server on /136.243.18.41:37181 2023-04-28 14:59:58,481 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase17.apache.org,37181,1682693812482 2023-04-28 14:59:58,481 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-28 14:59:58,481 ERROR [Listener at localhost.localdomain/40193-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@889ff5b rejected from java.util.concurrent.ThreadPoolExecutor@cb67a92[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2002] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-28 14:59:58,481 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-28 14:59:58,481 ERROR [Listener at localhost.localdomain/40193-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3f833cb5 rejected from java.util.concurrent.ThreadPoolExecutor@cb67a92[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2002] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-28 14:59:58,481 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase17.apache.org,37181,1682693812482] 2023-04-28 14:59:58,482 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase17.apache.org,37181,1682693812482; numProcessing=1 2023-04-28 14:59:58,483 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase17.apache.org,37181,1682693812482 already deleted, retry=false 2023-04-28 14:59:58,483 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase17.apache.org,37181,1682693812482 expired; onlineServers=0 2023-04-28 14:59:58,483 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase17.apache.org,37045,1682693811371' ***** 2023-04-28 14:59:58,483 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-28 14:59:58,483 DEBUG [M:0;jenkins-hbase17:37045] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3e73ddb0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase17.apache.org/136.243.18.41:0 2023-04-28 14:59:58,483 INFO [M:0;jenkins-hbase17:37045] regionserver.HRegionServer(1145): stopping server jenkins-hbase17.apache.org,37045,1682693811371 2023-04-28 14:59:58,483 INFO [M:0;jenkins-hbase17:37045] regionserver.HRegionServer(1171): stopping server jenkins-hbase17.apache.org,37045,1682693811371; all regions closed. 2023-04-28 14:59:58,483 DEBUG [M:0;jenkins-hbase17:37045] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 14:59:58,483 DEBUG [M:0;jenkins-hbase17:37045] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-28 14:59:58,483 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-28 14:59:58,484 DEBUG [M:0;jenkins-hbase17:37045] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-28 14:59:58,484 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster-HFileCleaner.large.0-1682693813475] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase17:0:becomeActiveMaster-HFileCleaner.large.0-1682693813475,5,FailOnTimeoutGroup] 2023-04-28 14:59:58,484 INFO [M:0;jenkins-hbase17:37045] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-28 14:59:58,484 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster-HFileCleaner.small.0-1682693813476] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase17:0:becomeActiveMaster-HFileCleaner.small.0-1682693813476,5,FailOnTimeoutGroup] 2023-04-28 14:59:58,484 INFO [M:0;jenkins-hbase17:37045] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-28 14:59:58,484 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-28 14:59:58,484 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-28 14:59:58,484 INFO [M:0;jenkins-hbase17:37045] hbase.ChoreService(369): Chore service for: master/jenkins-hbase17:0 had [] on shutdown 2023-04-28 14:59:58,484 DEBUG [M:0;jenkins-hbase17:37045] master.HMaster(1512): Stopping service threads 2023-04-28 14:59:58,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-28 14:59:58,485 INFO [M:0;jenkins-hbase17:37045] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-28 14:59:58,485 ERROR [M:0;jenkins-hbase17:37045] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-28 14:59:58,485 INFO [M:0;jenkins-hbase17:37045] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-28 14:59:58,485 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-28 14:59:58,486 DEBUG [M:0;jenkins-hbase17:37045] zookeeper.ZKUtil(398): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-28 14:59:58,486 WARN [M:0;jenkins-hbase17:37045] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-28 14:59:58,486 INFO [M:0;jenkins-hbase17:37045] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-28 14:59:58,486 INFO [M:0;jenkins-hbase17:37045] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-28 14:59:58,486 DEBUG [M:0;jenkins-hbase17:37045] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-28 14:59:58,487 INFO [M:0;jenkins-hbase17:37045] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-28 14:59:58,487 DEBUG [M:0;jenkins-hbase17:37045] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-28 14:59:58,487 DEBUG [M:0;jenkins-hbase17:37045] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-28 14:59:58,487 DEBUG [M:0;jenkins-hbase17:37045] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-28 14:59:58,487 INFO [M:0;jenkins-hbase17:37045] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.13 KB heapSize=393.89 KB 2023-04-28 14:59:58,583 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-28 14:59:58,583 INFO [RS:0;jenkins-hbase17:37181] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase17.apache.org,37181,1682693812482; zookeeper connection closed. 2023-04-28 14:59:58,583 ERROR [Listener at localhost.localdomain/40193-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@476ae857 rejected from java.util.concurrent.ThreadPoolExecutor@cb67a92[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2002] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-28 14:59:58,583 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): regionserver:37181-0x101d07a98b80001, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-28 14:59:58,583 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@605ff792] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@605ff792 2023-04-28 14:59:58,583 ERROR [Listener at localhost.localdomain/40193-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@f5bed3d rejected from java.util.concurrent.ThreadPoolExecutor@cb67a92[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2002] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-28 14:59:58,584 INFO [Listener at localhost.localdomain/40193] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-28 14:59:58,900 INFO [M:0;jenkins-hbase17:37045] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.13 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/5dadd77fef0e4c138868aa06110ab500 2023-04-28 14:59:58,904 INFO [M:0;jenkins-hbase17:37045] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5dadd77fef0e4c138868aa06110ab500 2023-04-28 14:59:58,905 DEBUG [M:0;jenkins-hbase17:37045] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/5dadd77fef0e4c138868aa06110ab500 as hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/5dadd77fef0e4c138868aa06110ab500 2023-04-28 14:59:58,909 INFO [M:0;jenkins-hbase17:37045] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5dadd77fef0e4c138868aa06110ab500 2023-04-28 14:59:58,909 INFO [M:0;jenkins-hbase17:37045] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44165/user/jenkins/test-data/cc7b0da7-a5d9-348f-53f9-e57d72dc5106/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/5dadd77fef0e4c138868aa06110ab500, entries=65, sequenceid=578, filesize=24.7 K 2023-04-28 14:59:58,910 INFO [M:0;jenkins-hbase17:37045] regionserver.HRegion(2948): Finished flush of dataSize ~345.13 KB/353414, heapSize ~393.88 KB/403328, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 423ms, sequenceid=578, compaction requested=false 2023-04-28 14:59:58,911 INFO [M:0;jenkins-hbase17:37045] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-28 14:59:58,911 DEBUG [M:0;jenkins-hbase17:37045] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-28 14:59:58,915 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-28 14:59:58,915 INFO [M:0;jenkins-hbase17:37045] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-28 14:59:58,916 INFO [M:0;jenkins-hbase17:37045] ipc.NettyRpcServer(158): Stopping server on /136.243.18.41:37045 2023-04-28 14:59:58,916 DEBUG [M:0;jenkins-hbase17:37045] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase17.apache.org,37045,1682693811371 already deleted, retry=false 2023-04-28 14:59:59,017 INFO [M:0;jenkins-hbase17:37045] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase17.apache.org,37045,1682693811371; zookeeper connection closed. 2023-04-28 14:59:59,017 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-28 14:59:59,018 ERROR [Listener at localhost.localdomain/40193-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@64730ab9 rejected from java.util.concurrent.ThreadPoolExecutor@278338c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3352] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-28 14:59:59,018 DEBUG [Listener at localhost.localdomain/40193-EventThread] zookeeper.ZKWatcher(600): master:37045-0x101d07a98b80000, quorum=127.0.0.1:61762, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-28 14:59:59,018 ERROR [Listener at localhost.localdomain/40193-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@26017769 rejected from java.util.concurrent.ThreadPoolExecutor@278338c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3352] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-28 14:59:59,019 WARN [Listener at localhost.localdomain/40193] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-28 14:59:59,023 INFO [Listener at localhost.localdomain/40193] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-28 14:59:59,139 WARN [BP-1686677835-136.243.18.41-1682693808338 heartbeating to localhost.localdomain/127.0.0.1:44165] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-28 14:59:59,139 WARN [BP-1686677835-136.243.18.41-1682693808338 heartbeating to localhost.localdomain/127.0.0.1:44165] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1686677835-136.243.18.41-1682693808338 (Datanode Uuid 72d0f6de-cc10-4518-ad17-bf64b7a974d3) service to localhost.localdomain/127.0.0.1:44165 2023-04-28 14:59:59,142 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c847dfce-b710-d479-c833-1a4b351fc988/cluster_0c9022ae-d6e9-07a9-5c8b-552302ea3460/dfs/data/data1/current/BP-1686677835-136.243.18.41-1682693808338] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-28 14:59:59,142 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c847dfce-b710-d479-c833-1a4b351fc988/cluster_0c9022ae-d6e9-07a9-5c8b-552302ea3460/dfs/data/data2/current/BP-1686677835-136.243.18.41-1682693808338] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-28 14:59:59,182 INFO [Listener at localhost.localdomain/40193] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-04-28 14:59:59,339 INFO [Listener at localhost.localdomain/40193] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-28 14:59:59,482 INFO [Listener at localhost.localdomain/40193] hbase.HBaseTestingUtility(1293): Minicluster is down